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ABSTRACT 


The U.S. Marine Corps developed the Combat Active Replacement Factor (CARF) 
methodology as a way to obtain reliable logistics planning factors to aid in the estimation 
of equipment losses in future conflicts. The continuous evaluation and validation of these 
types of methodologies is considered of critical importance, since its effects directly 
impact combat effectiveness, supply chain management, logistics, acquisitions, and 
overall budgeting. This thesis analyzes a proposed methodology for use in calculating 
Explicitly Calculated CARFs (ECCs), making use of real-world Master Data Repository 
(MDR) data from previous low- and medium-intensity conflicts. As well, this thesis 
analyzes proposed regression models used in calculating Federal Supply Code (FSC) and 
Federal Supply Group (FSG) CARFs. We employ bootstrapping techniques in order to 
analyze the sensitivity of ECCs and find that as many of 70% may exhibit extreme 
sensitivity to reasonable changes in usage data. We employ Ordinary Least Squares 
regression models to estimate CARFs by FSC and FSG and obtain dramatically more 
CARFs relative to the draft methodology. Finally, a cross validation of a sample of the 
regression models reveals that CARFs generated from such models tend to vary 
substantially from their actual values. 
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EXECUTIVE SUMMARY 


The Combat Active Replacement Factors (CARF) are logistics-planning factors that aid 
in the estimation of equipment losses in future conflicts. The continuous evaluation and 
validation of this methodology is critically important, since the CARF values are 
employed in the War Reserves System (WRS), where they directly impact combat 
effectiveness, supply chain management, logistics, acquisitions, and overall budgeting. 
Evaluating the effectiveness of these procedures is a meaningful area of study. 

We begin with a comparative analytical review based on a sensitivity analysis of 
the currently proposed methods. Explicitly Calculated CARFs (ECCs) are factors 
calculated using available historical data, in this case, from Operation Iraqi Freedom and 
Operation Enduring Freedom, by observing the quantities of equipment deployed and the 
reported losses of such equipment during the years 2005 through 2010. We generate 50 
bootstrapped replications of the historic usage data to obtain a distribution of ECCs for 
those items uniquely identified by Table of Authorized Material Control Numbers 
(TAMNs) with available usage data. We show that based on the individual level and 
phase of conflict, many ECCs are significantly sensitive to minimal changes in the data 
with which they are calculated. We find that 12 of the low level ECCs and 140 of the 
medium level ECCs are not significantly different from zero. The least sensitive were 
Assault ECCs for Low Intensity Conflict (ECC LA), where 82 of 611 showed high 
sensitivity, while the most sensitive were Sustainment ECCs for Medium Intensity 
Conflict (ECC_MS), with 357 of 514 being highly sensitive. 

The proposed methodology for calculating CARFs for equipment that do not have 
all the necessary information to obtain an ECC involves building linear regression models 
based on the equipment’s classification within its hierarchy of the Federal Supply Code 
(FSC) and Federal Supply Group (FSG). These CARFs are denoted CRCs and GRCs, 
respectively. This leverages the assumption that similar end items, within the same 
classification, have similar attributes that can be used to calculate their attrition rate. We 
employ Ordinary Least Squares (OLS) regression and achieve vast improvements in 
tenns of significance and model validity across the studied levels and phases of conflict, 
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relative to the draft methodology and we obtain a substantial increase in the number of 
CRCs and GRCs produced. For example, our method obtains 369 CRCs for the Assault 
phase of Medium Intensity Conflict (CRC MA) and 398 GRCs for the Assault phase of 
Low Intensity Conflict (GRCLA), relative to 2 and 46 according to the draft 
methodology. 

We also cross-validate a sample of the regression models, with the primary intent 
to evaluate the performance of the regression fits and the hope of gaining some insight as 
to determining how sensitive the models would be to gaps in the data. In every trial, we 
withheld 20%, 15%, and 10% of the available data. With this, we could run the 
regression models and compare estimated CRC (or GRC) with the TAMCN’s actual ECC 
as well as the bootstrapped Confidence Intervals. The examples we used here were the 
FSCs 2320 and 5820, and the FSG 23. These FSCs and FSG were chosen because of their 
amount of TAMCNs with available data and because of their primary importance, as 
rolling stock and communications equipment, to the operating forces. We find that even 
withholding 10% of the data upon which the original model was built produces results 
that tend to fall outside even the bootstrapped CIs. 

Intending to provide an overall methodology that can predict over 6,000 CARF 
values based on the resulting values of the limited information on just 600 CARFs might 
not be the best approach. Though no validation was performed in the entire OLS process 
applicability and results, the conclusions obtained demonstrate the need for further 
research into proper methodologies. 
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I. INTRODUCTION 


A. MISSION AND PURPOSE OF THIS THESIS 

This study’s intent is to provide a comparative analytical review, based on a 
sensitivity analysis, of the proposed methods for forecasting equipment losses in future 
conflicts, known as Combat Active Replacement Factors (CARFs). This thesis, in a very 
narrow scope, will also aid in the future research of various proposed methodologies by 
reviewing multiple regression models that can be used in CARF value calculations. 

The United States Marine Corps developed the CARF methodology as a way to 
obtain reliable logistics planning factors to aid in the estimation of equipment losses in 
future conflicts. The continuous evaluation and validation of this methodology is 
important since the CARF values are employed in the War Reserves System (WRS) and 
where they directly impact combat effectiveness, supply chain management, logistics, 
acquisitions, and budgeting. 

CARFs are considered critical elements in the requirements determination process 
due to their effect on the entire Marine Corps’ Acquisition Program. Funding 
requirements can be substantially altered with a small change in the CARF of either a 
“high-density or high-value dollar item” (Department of the Navy, 2010a, p. 1-5). The 
continued need to review the effectiveness of the procedures that derive these combat 
planning factors is a very attractive and meaningful area of study. 

This thesis assesses the sensitivity of the results of the process for producing 
CARF estimates to reasonable changes in observed usage data. In addition, it maps out 
an alternative method for generating CARFs for items that lack usage data and it 
demonstrates a technique for evaluating the effectiveness of that method. 

B. BACKGROUND 

The CARFs have a crucial role in estimating the Marine Corps’ War Reserve 
Materiel Requirements (WRMR) and feed into the Approved Acquisition Objectives 
(AAO). It is the responsibility of the Deputy Commandant for Installations and Logistics 
to validate CARFs and the methodology through which they are obtained, as per Marine 
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Corps Order (MCO) 5311.ID. CARFs reflect the anticipated combat replacement needed 
for equipment losses, on a 30-day basis, incident to amphibious operations and other 
combat operations (Department of the Navy, 2010b). As per MCO 5311.ID, a unit is 
considered to be in combat operations only when in direct enemy contact. Accurate 
CARFs provide the planning flexibility to meet requirements of any conflict, by theater 
and specific operational plan. 

CARFs are an indispensable tool in detennining War Reserve Materiel (WRM) 
stocks. NAVMC 4000.1 specifies that CARFs be established for Type 1 Principal End 
Items (Table of Authorized Material Control Number [TAMCN]) beginning with the 
letter A, B, C, D, or E) designated with a Combat Essential Code 1 (CEC 1), for Class II, 
clothing and individual item supplies, and Class VII, principal end item supplies, 
following the Plans, Policies, and Operations (PP&O) publication that contains the 
essential equipment list (Department of the Navy, 2010b). CARFs also have a direct 
impact on the strategic lift and sustainment requirements for a deploying Marine Air- 
Ground Task Force (MAGTF). These replacement factors are the planning guidelines 
used to properly equip forces in the specific phases, level, and duration of a conflict, in 
order to maintain quantities up to their Table of Equipment (T/E) level in the event that 
essential materiel is damaged beyond utilization during a contingency operation 
(Department of the Navy, 2010a). 

CARFs are categorized in two phases of combat operations. The Assault Phase, 
represented by the first 30 days of combat, and the Sustainment Phase, represented by 
every subsequent 30-day period. The historical data used in calculating CARFs is, under 
a draft methodology proposition, categorized into three levels of conflict intensity: 

• Low-intensity conflict (LIC): This level of intensity is described as 
a political-military confrontation between contending states or 
groups below conventional war and above the routine, peaceful 
competition among states. LICs are often localized, generally in 
third-world countries, but contain regional and global implications. 

A LIC can range from an insurrection to a more organized use of 
employing political, economic, informational, and military 
instruments, to include irregular warfare scenarios. 

• Medium-intensity conflict (MIC): This level of intensity is 
perceived as a “protracted” employment of regular annies in 
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combat and a major manifestation of power by the opposing and 
responding nations, with the designation and prioritization of 
military objectives in order to achieve political and economic 
goals. 

• High-intensity conflict (HIC): This level of intensity is described 
as a conflict in which we observe a relatively unconstrained use of 
power by one or more nations to gain or protect territory and 
interests that directly affect the survival of the nation. Extreme 
levels of violence and the employment of the full range of military 
forces are evident in this level of conflict (Draft version of a 
contracted 2011 study [Draft CARF Technical Report, 2011], 3). 

The various CARFs to be obtained are shown in Table 1-1. 


Table 1-1. CARF Values by Conflict Intensity and Phase (Draft CARF, 2011) 


Intensity of Conflict 

Assault Phase 

Sustainment Phase 

High Intensity Conflict 
(HIC) 

CARF High Intensity Assault 
(CARFHA) 

CARF High Intensity 
Sustainment (CARFHS) 

Medium Intensity 
Conflict (MIC) 

CARF Medium Intensity 
Assault (CARFMA) 

CARF Medium Intensity 
Sustainment (CARFMS) 

Low Intensity Conflict 
(LIC) 

CARF Low Intensity Assault 
(CARFLA) 

CARF Low Intensity 
Sustainment (CARFLS) 


C. LITERATURE REVIEW 

In early 1985, Professor Lindsay of the Naval Postgraduate School published a 
technical report titled An Examination of the USMC Combat Active Replacement Factor 
(CARF) Determination System. This report mentions the wide ongoing efforts that were 
conducted throughout the early 1980s to develop new procedures for the Marine Corps’ 
CARF values generation, since until that decade only adapted Army replacement factor 
values were used. In his report, Lindsay compares two methods of estimating CARF 
values. 

The first method relies on scenario-oriented models using mean-time-to-loss 
(MTTL) estimates and means of the observed distributions in different situations where 
specific items are subject to either the same loss rate or varying loss rates throughout a 
time period. Lindsay’s report emphasizes that the use of exponential distributions to 


3 




arrive at a time to loss is based on the same assumption of constant failure rates employed 
in reliability theory and, at that time, “essentially mandated” by the Department of 
Defense for this type of study (Lindsay, 1985). 

The second method in this analysis looks at arriving at a CARF value based on 
the professional military judgment of many experts who rate “chance of loss” for specific 
types of equipment. Through this approach, a panel of subject matter experts (SMEs) is 
tasked with ranking the expected likelihood of loss of an item in various situations, and 
then compares which items should have a higher or lower CARF value. Those groups of 
rankings are converted into an interval scale, which can provide a scaled average score 
for each item and, in doing so, can provide CARF estimated values. As mentioned in 
Lindsay’s report, the disadvantage observed with this methodology is the large number of 
SMEs needed in order to provide a sensible level of probability of success since it is 
“based upon the disagreement among judges” (Lindsay, 1985, p. 23). In his conclusions, 
Lindsay points out that, in certain instances, CARF values that are generated by relying 
on SME input might be the preferred approach since they often offer more substantial 
empirical evidence for the reasoning of the resulting values and the way they are 
obtained. 

In December 1985, Major Hee Sun Song’s (Republic of Korea Army) presented 
the results of his thesis research titled “Application of Life Distributions Estimated 
Equipment Losses in Combat,” in which he proposed a similar CARF generation process 
based on MTTL estimates that can be applied to various types of life distributions, such 
as exponential, Weibull, or gamma, as well as to a nonhomogeneous Poisson process 
(Song, 1985). He suggests that CARFs generated is this manner are more sensible 
because it is easier to understand the thought process that brings about such input values 
and results. 

The results of Song’s thesis postulates that the intention to consider and compare 
all of these life distributions in the study of life expectancy of an item in a combat 
scenario aids in the proper selection of such a distribution, when one cannot be exactly 
identified because of the complexity of war scenarios in itself. Song also emphasizes that 
his work serves as an extension of Lindsay’s work by “ removing the need to assume a 
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constant loss rate.“ (Song, 1985, p. 43) and suggests that it might be an easier approach if 
we consider different levels of combat intensity and calculate MTTLs for each with the 
intention that we avoid “sealing” all our estimations with one over-all-population MTTL. 
But, of note, in his conclusion he also states, “The choice of scenario and life distribution 
will, of course, depend primarily on the specific item being studied” (Song, 1985, p. 43). 

In 1990, emphasizing the importance of having properly generated CARF values 
and their effects in the war reserves stock, Major Joseph L. Stylons, United States Marine 
Corps (USMC), publishes his findings in an NPS thesis titled “War Reserves Stocks and 
Marine Corps Sustainability.” He takes the approach of explaining how the “deficit-in- 
assertiveness” of our planning factors infect the proper war reserve stocks, the entire 
acquisitions program, and consequently the overall ability of the Marine Corps to project 
power as a global military force. The business point of view of this thesis reflects the 
wide ramifications of the matter of study to the overall funding levels and acquisition 
power and a definite need for emphasis in the use of these logistics planning factors 
(Stylons, 1990). 

A review of the literature failed to identify any studies more recent than those 
discussed above, to include studies concerning other services. 

D. AN EARLY VERSION OF A PROPOSED METHODOLOGY AND 

AVAILABLE DATA 

A draft version of the study of a proposed Fiscal Year (FY) 2011 CARF 
assignment methodology follows a specific assignment algorithm decision flow map 
depicted in Figure 1-1. 
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Figure 1-1. CARF Assignment Algorithm Decision Flow (LSX 2011). 
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This decision flow map encompasses a total of almost 7,300 calculated CARFs 
using different statistical models. Of the total CARFs, 15% are calculated by following 
an explicitly calculated CARF methodology and 14% are calculated using varying forms 
of regression models (Logistics Operations Research Branch [LSX], 2011). 

In CARF assignment methodology, ECC values are calculated for those items, 
referred to as TAMCNs, with recorded data obtained from Operation Iraqi Freedom 
(OIF) and Operation Enduring Freedom (OEF) losses in low- and medium-intensity 
conflicts. Because of the lack of data, high-intensity conflicts require the use of combat 
modeling data and are outside of the scope of this thesis. This level of conflict is not 
considered in the methodologies reviewed in this study. 

The two-phase categorization determines the inclusion of a Median Maintenance 
Deadline (MMD) factor added when calculating CARFs within the assault phase only, 
accounting for the temporary losses due to maintenance during this initial phase. This 
added factor is the arithmetic mean of the deadline time, per TAMCN, obtained from 
historical data from OIF and OEF (LSX, 2011). 

In order to calculate a CARF over an assault or sustainment phase, for a specific 
TAMCN, the following essential pieces of infonnation are needed: the number of days in 
the duration of the deployment, the total number of items deployed to the theater, the 
total number of items destroyed by date, and, for the assault period, the MMD. For 
example, using the equations first developed by the Marine Corps Combat Development 
Command, Operations Analysis Division, and described in the “Draft Combat Active 
Replacement Factors (CARFs) Technical Report” draft version of a 2011 study (LSX, 
2011) a CARF calculated for the assault phase is obtained as follows: 


Assault CARF = 


KIN 

T 



+ MMD 


where: 

K = Total number permanently lost 
N = Total number of items deployed 

T = Total number days TAMCN was deployed to the contingency operation, and 
MMD = Median Maintenance Deadline 
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In a CARF calculated for the sustainment phase, the following equation is used 
(LSX, 2011): 


Sustainment CARF = 


'KIN 


\ 

x 30 days 


ECC methodology is the most accurate and preferred approach to calculating 
CARFs. Currently, 1,109 CARFs have been calculated using this ECC method (LSX, 
2011 ). 

For TAMCNs without sufficient information to calculate an ECC, the next step in 
the decision flow map is to calculate their CARFs by building a regression model from 
items within their same Federal Supply Code (FSC) that already have ECCs. These 
supply codes, and Federal Supply Groups (FSGs), represent National Stock Numbers 
(NSN), which are 13-digit numbers used by the Federal Government and assigned by the 
Defense Logistics Agency Customer Interaction Center (DLA CIC) to identify and 
classify products. For example, the first four digits of an NSN represent these FSCs and 
FSGs, which would contain similar groups of end items, i.e., NSN 2355-XX-XXX- 
XXXX represents the group of wheeled combat, assault, and tactical vehicles. 

The type of model for estimating CARFs by FSC and FSG that we examine in 
this thesis is linear regression. However, it is important to note that the activity 
responsible for producing CARFs no longer uses linear regression as their preferred 
method. As specified in the Detailed CARF Assignment Methodology and CARF-STAT 
Documentation report (LSX, 2011) and in accordance with the legacy FY2011 CARF 
Assignment Algorithm, nested multivariate analysis techniques were used. Within the 
regression, two numerical predictors, total approved allowance and cost for a TAMCN, 
are selected as the most statistically significant and influential factors, based on the 
available data and the regression model that initially includes 23 terms. In this study of a 
proposed methodology, the resulting R-square obtained through this regression is 0.9843 
(LSX, 2011), reflecting a sufficient goodness-of-fit for calculated CARFs. 

For CARFs that cannot be calculated under a CRC method, the regression model 
is built on the next higher echelon of classification, the FSG. The FSG Regression CARF 
(GRC) method uses the attributes of the TAMCNs for which CARFs were obtained 
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through the previous two higher levels, FSC and ECC. Once again, within the regression, 
the most statistically significant two numerical predictors remain the total approved 
allowance and the cost for a TAMCN (LSX, 2011). 

E. PRELIMINARY OBSERVATIONS AND RECOMMENDATIONS ON 

HOW TO PROCEED 

As a first approach, we bootstrap the historical usage data in order to perform a 
sensitivity analysis for the calculation of ECCs. We develop a criterion that describes the 
sensitivity of ECCs to changes in the usage data. We find that as many as 140 TAMCNs 
have ECCs so sensitive that their bootstrapped confidence intervals contain 0.0. In 
addition, we find that as many as 303 low-level conflict ECCs and 357 medium-level 
conflict ECCs that have valid confidence intervals may be overly sensitive. 

As a second objective, we focus our efforts on the 1,137 CARFs in which 
FSC/FSG Regression CARF methodology is employed, by assessing the validity of each 
regression model presented. We are able to produce 14,292 CARFs using different 
regression models. In addition, we outline a method for assessing the validity of these 
CARF estimates. 

Ultimately, our results show that caution is in order when applying statistical 
methods in an attempt to overcome the pervasive gaps in usage data. Given the relative 
paucity of usage data, and the sensitivity of parameters calculated from it, it is probably 
wise to consider at least augmenting the process with combat modeling and simulation 
and the contributions of subject matter experts. 
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II. AVAILABLE DATA AND DETAILED CARF REGRESSION 

METHODOLOGY 


A. PROVIDED DATA AND ITS EMPLOYMENT 

The data for this study are contained in a table that covers information on 6,137 
unique TAMCNs. As mentioned in a draft version of a contracted 2011 study (LSX, 
2011), the JMP statistical analysis toolset (see www.jmp.com) is used because of its 
commercial off-the-shelf availability and its ability to be easily employed for quick 
analysis and create multiple regression models. This file includes scripts for 
automatically calculating ECCs for low- and medium-intensity conflicts, to include the 
respective assault phases, based on usage data from Marine Expeditionary Forces (MEF) 
between 2005 and 2010 in OIF and OEF. The file also includes scripts for fitting FSC 
and FSG regression models, as well as further-on procedures that are depicted in the 
decision flow map, such as Average FSC CARF (ACC) and Weighted Average FSG 
CARF (WGC), which are not within the scope of this thesis. 

The 6,137 rows represent each TAMCN, or individual equipment variant, that the 
MEFs employ, and capture a wide variety of information, such as dimensions, weights, 
commodity, nomenclature, NSNs, CECs, and weapons systems codes (see Appendix A 
for full data sample). 

1. Table of Authorized Material Control Number (TAMCN) 

The TAMCNs are the most expedient way to track the equipment variants while 
maintaining data integrity with the MDR source files provided. TAMCN5s only show the 
first five numbers of every one of these TAMCNs, which provides a grouping of similar 
TAMCNs. The first letter of a TAMCN refers to the commodity code, while the 
nomenclature (NOMEN) captures the assigned descriptive name of the equipment 
variant. 


2. Combat Essential Codes (CECs) 

CECs are designator numbers that indicate if the item is essential in a combat 

operation. As described in Chapter I, these CECs are assigned to equipment variants that 
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have been identified as Combat Essential End Items and are classified as Type 1, Type 2, 
or Type 3. Type 1 end items are TAMCNs beginning with A, B, C, D, or E. The WMR 
Handbook states that when calculated CARFs are employed, they are only applied to 
“TAMCNs associated with Type 1 End Items for Class II and VII” (Department of the 
Navy, 2010a, et al., p. 1-5). In the data presented, only 2,421 items are combat essential, 
represented by a CEC number of “1,” and 3,716 items are represented by a CEC number 
of “0” as noncombat essential. 

3. Total Approved Acquisition Objective (AAOTOTAL) 

The total equipment approved quantities are captured under AAO TOTAL, which 
is an aggregate of the quantities reported by each of the three MEFs, the WRM reserves, 
each of the worldwide regional Marine Forces Commands, and the three Maritime 
Prepositioning Squadrons (MPSs). 

4. Equipment Variant Best Cost (BESTCOST) 

The unit price of these equipment variants is also reported in this data file under 
the header BEST COST, which is the greater value when comparing three costs in the 
data file: the cost to procure (COSTPROCURE), the cost to replace 
(COST REPLACE), and the standard unit price (STD UNIT PRICE). 

5. Dimensional Information 

For most TAMCNs, physical dimensions such as weight, length, nonnal height 
and lowest height (i.e., motorized cranes), and width are included in the dataset all of 
these are factors that could be observed as possibly influential in a regression model. 

6. Total Quantity On-Hand and Losses (OH and WIR) 

Historical data, such as total quantity on-hand (deployed) and total losses reported 
from 2005 through 2010, are available for approximately 10% of the TAMCNs. The 
on-hand data available is titled OH_VII_MEF_20XX, in which the last digits refer to the 
year it covers. The dataset only provides observations for 908 TAMCNs for each of the 
six years of recorded data. The equipment losses are also accumulated by year under the 
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headers WIRVIIMEF20XX and comprise a total of 984 available observations per 
year. These totals refer to the number of cells that contain information for the specific 
TAMCNs for which data is available, i.e., only reported equipment on-hand during that 
year and only reported losses, if any for that year, will appear. 

The data in the total quantity on-hand and losses is employed in the calculation of 
ECCs, as described in Chapter I, and fonn the core of the first analytical steps of this 
thesis for the purpose of bootstrapping over the losses, while maintaining the total on- 
hand data constant. This allows multiple replications that conform to the characteristics 
of the original ECC data, while obtaining simulated losses to further study performance 
of the methodology. This process is further detailed in Chapter III. 

Emphasis is placed on analyzing this first step of calculating ECCs. Since the 
calculated ECC will serve as a “seed” from where the rest of the decision “tree” is built 
upon, it is of keen importance to prove resiliency. We obtain a confidence interval that 
reflects how much we can rely on this initial step, which will grow the different 
“branches” that become the FSC, FSG, ACC, and WGC CARFs. 

B. DESCRIPTION OF THE REGRESSION MODELS AND OBSERVATIONS 

As detailed in Chapter I, a proposed CARF methodology uses regression analysis 
to build a predictive model in which CARF values are calculated for those individual 
equipment variants that do not have enough historical information for an ECC approach. 

Equipment variants that fall within the same general end use are classified under 
the same FSC. Within each FSC, many TAMCNs may not have historical usage data 
available. The intent is to fit a regression to the TAMCNs within the FSC for which 
historical data is available, and use that model to predict a CARF for the TAMCNs that 
lack usage data. As with all regression models, the bigger the N, where N = number of 
TAMCNs with historical data, the more reliable and/or acceptable prediction we expect 
to encounter. 

Both regression models assume that FSC and FSG assignments have been 
validated by the Marine Corps Combat Development Command (MCCDC) Integration 
Division (ID) and considered accurate (LSX, 2011). This is of relevance since the 
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similarity between these assigned FSC and FSG classifications is something from which 
we can draw similarities in order to generalize the fitting of the employed regression 
models. 

The 6,137 TAMCNs are classified in a total of 274 FSCs, which are categorized 
under a total of 63 FSGs. The breakdown of the total number of TAMCNs per FSG is 
presented in Table 2-1. 


Table 2-1. 


FSG 

N 

10 

128 

12 

50 

13 

42 

14 

23 

15 

4 

16 

76 

19 

11 

23 

361 

24 

7 

25 

57 

26 

1 

28 

6 

29 

5 

34 

38 

35 

32 

36 

12 


Total Number (N) of TAMCNs by FSG 


FSG 

N 

37 

12 

38 

87 

39 

40 

40 

15 

41 

102 

42 

225 

43 

39 

44 

2 

45 

10 

46 

22 

47 

6 

49 

186 

51 

180 

52 

9 

53 

5 

54 

55 


FSG 

N 

56 

5 

58 

822 

59 

123 

60 

6 

61 

144 

62 

25 

63 

20 

65 

64 

66 

483 

67 

35 

68 

12 

69 

58 

70 

340 

71 

30 

72 

24 

73 

26 


FSG 

N 

74 

16 

75 

19 

76 

1 

79 

15 

80 

1 

81 

41 

83 

55 

84 

872 

85 

2 

88 

5 

89 

45 

91 

1 

93 

8 

95 

5 

99 

388 


An example of the use of these FSGs and FSCs, in a simplified view, is provided 
in Table 2-2. In the example, FSG 15 refers to the overall group of aircraft and airframe 
structural components. FSC 1550, drones, falls under this grouping category and includes 
four distinct TAMCNs, as presented in the table. The complete list of FSCs is presented 
in Appendix B. 
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Table 2-2. Example of FSG 15, FSC 1550, and TAMCNs Classification 


FSG 

FSC 

TAMCN 

NOMENCLATURE 

15 



AIRCRAFT AND AIRFRAME STRUCTURAL COMPONENTS 


1550 


DRONES 



A01217G 

MICRO UNMANNED AERIAL VEHICLE (MUAV) 



A03217G 

DRONE ACFT, RAVEN DDL 



A32527G 

UAV SYSTEM, DRAGON EYE 



A32527G 

UAV SYSTEM, RAVEN 


The regression model employed for the FSC CARF uses a Generalized Finear 
Model (GFM) regression with a Poisson distribution and a log-link function. The 
AAOTOTAF and BESTCOST, and their first order interaction, are the regressors for 
each FSC. The regression models employed for the FSG CARF replicate this same 
procedure. 

The early version draft CARF report justifies the use of the GFM regression by 
stating that its primary benefit is the ability to employ a “variety of distribution families 
and associated fit functions” (FSX 2011, pg 9) in order to capture the complexity of 
dependencies between the necessary regression factors. While unable to re-create the 
same results as presented by the draft CARF report, analyzing the results obtained when 
following the regression methodology reveals that none of these FSC regressions (CRCs) 
are valid. Most of them show an insufficient chi-square test result and—a most 
compelling observation—all of the parameter estimates’ p-values are at or above a 0.9, 
indicating no real importance as far as being influential factors in the regression. 

The decision flow map from the proposed CARF methodology, presented in 
Chapter I, indicates that previously obtained calculations for the FSC CARF, in 
conjunction with the previously calculated ECCs, are to be used in the FSG CARF 
regression models. This is not evident in the FSG CARF computational procedure 
and regression analysis script provided. In fact, the procedure is repeated and directly 
follows the same steps used in calculating FSC CARFs, regressing on AAO TOTAF 
and BESTCOST and their first order interaction, the cross effect 
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AAO_TOTAL*BEST_COST, without making use of the successfully calculated ECCs 
and FSCs. This, in turn, would neglect the value added to the FSG regression as provided 
by previously calculated FSCs. 

C. VISUALIZING CURRENT RESULTS OF THE PROPOSED CARE 
METHODOLOGY 

The total count of CARFs assigned, by level and phase of conflict, using the ECC, 
CRC, and GRC procedures are depicted in Table 2-3. From the total of 6,137 TAMCNs, 
and only reporting on the 2,421 CEC 1 TAMCN, on average almost 31% of them have 
been assigned an ECC CARF; less than 5% have been assigned a CRC CARF; and less 
than 3% have been assigned a GRC CARF. 


Table 2-3. Total CARFs Assigned with % of Total (ECC/CRC/GRC) 



ECC 

% of Total 

CRC 

% of Total 

GRC 

% of Total 

CARFLA 

611 

30.40 

190 

9.45 

46 

2.29 

CARF LS 

626 

31.21 

197 

9.82 

51 

2.54 

CARF MA 

611 

30.40 

2 

0.10 

59 

2.94 

CARF MS 

621 

30.91 

3 

0.15 

65 

3.24 


Figures 2-1 through 2-4 give a graphical representation of the number of CARFs 
assigned by all the procedures used in a proposed CARF methodology. 
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Figure 2-1. Obtained LA CARFs by Source. 
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Figure 2-2. Obtained LS CARFs by Source. 
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Figure 2-3. Obtained MA CARFs by Source. 
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Figure 2-4. Obtained MS CARFs by Source. 

Figures 2-1 and 2-2 depict the sources for the low-level conflict, the assault and 
sustainment phases, respectively. For the assault phase of this level of conflict over 9% of 
the CARFs are obtained by a CRC approach, while roughly 2% is obtained through a 
GRC approach. We can observe comparable results for the sustainment phase, where 
close to 10% is obtained by CRC and only 2.5% is obtained by GRC. The perception of 
this difference can be simply explained by understanding that the sustainment phase 
comprises more observed time and more data captured during that event. 

In contrast, Figures 2-3 and 2-4 depict the sources for the medium-level conflict 
and show the overall inverse results. For the assault phase, a mere 0.10% is obtained 
through CRC, while 3% is obtained through GRC. Similarly, for the sustainment phase, 
0.15% is obtained through CRC compared to over 3% by GRC. We can interpret this 
difference as a result of the evident contrast between a medium-level conflict, which has 
historically been of a shorter duration, and a low-level conflict. In this case, the 
scarceness observed in the available data in a medium-level conflict forces the 
employment of the FSG regression models (GRC) to be more frequently relied upon for 
CARF value calculations, thus generalizing loss quantities within an FSG. 

Overall, the intention of increasing reliance on the regressed models employed in 
calculating CARF values arises from (1) the sparseness of the available historical data 


18 







used in calculating ECCs, which demands a method that can fill those “gaps” with 
reliable CARF values; and (2) that most of the CARF values that are not currently 
obtained from a explicitly calculated approach, using the sparse data available, are 
obtained from other methodologies that depend upon the resiliency of the CRC and GRC 
methodology, as is the case of an ACC and the GRC (LSX, 2011). If historical data 
collection and management is bolstered, the regressed methodology can be further 
developed to serve as bedrock for a more resilient and accurate means of calculating 
equipment needs for the operating forces and overall acquisitions policies. 
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III. BOOTSTRAPPING ON ECC CARF AND REGRESSION 
ANALYSIS OF FSC AND FSG CARF MODELS 


A. BOOTSTRAPPING ON AVAILABLE HISTORICAL DATA FOR ECC 

SENSITIVITY ANALYSIS 

CARFs are essentially an expected number of casualties in a future conflict per 
unit of equipment deployed in combat. The dataset contains the annual total equipment 
deployed and total casualties observed for approximately 740 TAMCNs. We obtain 
ECCs for each CARF for which there is usage data. Since the data are sparse and the 
proportion of casualties small, the ECCs may be highly sensitive to reasonable variation 
in the data. We test the extent to which the ECCs are overly sensitive to changes in the 
input data by bootstrapping 50 additional datasets. Unless otherwise indicated, all 
TAMCNs refer only to CEC 1 items. 


1. Bootstrapping 

For a given TAMCN with a particular number of units deployed to a combat 
zone, the number of casualties can be modeled as a binomial distribution. Each day 
represents an opportunity for each piece of equipment to experience a casualty, thus 
complying with the assumption of a fixed number of trials. We assume independence 
between trials and constant probability of failure. Thus, the number of casualties for a 
given TAMCN for a particular year is distributed binomially, as follows: 


f 


casualties ~ B1NOM 


(i n = Totalonhand x 365days). 


V 


A 


x 
njj 


where x is the total number of casualties observed for that year and 365 days of the year 
representing 365 opportunities for the total deployed number of TAMCNs to fail. 

In perfonning this sensitivity analysis, the bootstrapping methodology is applied 
to the equipment losses (WIR_VII_MEF_20XX) for the years 2005 thru 2010. These 
data are extracted and placed in a Microsoft Excel file to be used as an input file for a 
JAVA script to more efficiently perform the necessary functions, using a random number 
generator to obtain the results. Fifty replications of this procedure are perfonned 
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generating a total of 306,850 rows of bootstrapped data available once all replications are 
stacked. We then run the original ECC script on the bootstrapped dataset in order to 
obtain a distribution of ECCs for each TAMCN. 

We obtain a 90% bootstrapped confidence interval for each TAMCN that was 
originally assigned an ECC. We observe the following: Of the 611 ECCLAs and 611 
ECCMAs, 12 TAMCNs have bootstrapped confidence intervals that contain 0.0. Of a 
total of 626 ECCLSs, 135 have bootstrapped confidence intervals that contain 0.0. Of 
the total of 626 ECCMSs, 140 have bootstrapped confidence intervals that contain 0.0. 

We are able to identify the problem that creates these results by observing that in 
the numbers reported in the original file presented for the study of this thesis, the 
casualties reported under the column headers WIR_VII_MEF_2XXX, as defined in 
Chapter II, have very sparse data. The lack of infonnation for losses in these specific 
TAMCNs, throughout the years reported, create a problematic “noise” that can affect the 
calculations used in producing valid ECCs and regression predicted values. In 
Table 3-1, the 12 ECC LAs and MAs that have a lower bound of 0.0 are presented 
together with their cost, emphasizing the impact unreliable ECC values can have for 
some of these TAMCNs. 


Table 3-1. FSCs with ECC LAs and MAs with 90% Bootstrapped Confidence 

Intervals that Contain 0.0 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

Item 
Cost ($) 


A70467G 

GENERATOR, SIGNAL 

0 

0.0036 



A70467G 

GENERATOR, SIGNAL 

0 

0.0040 



A70467G 

40 GHZ SIGNAL GENERATOR 

0 

0.0040 



A70527G 

1 GHZ SIGNAL GENERATOR 

0 

0.0011 



A70597G 

SOIL RESISTIVITY TESTER 

0 

0.0004 



A70847G 

LOCAL AREA NETWORK TEST SET 

0 

0.0019 

7,116 


A70847GA 

ANALYZER, NETWORK 

0 

0.0021 

10,481 

8110 

B05717B 

DRUM, FABRIC, COLLAPSIBLE, 500 GAL. CAP. 

0 

0.0012 


8110 

B05717B 

DRUM, FABRIC, COLLAPSIBLE, 500 GAL. CAP. 

0 

0.0011 


5180 

B22602E 

TOOL KIT, PIONEER, ENGR SQUAD 

0 

0.0007 


5180 

B22602E 

PIONEER KIT (SQD) 

0 

0.0009 


4933 

E05002E 

KIT, GAUGE, PULLOVER, COMPLETE 

0 

0.0037 

2,927 
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In Table 3-2, only the 10 highest-cost items for ECCLS and ECCMS with a lower 
bound of 0.0 are presented. The complete list of these particular TAMCNs is presented in 
Appendix C. Also, their cost is included in Table 3-2 to emphasize the severity that these 
inconsistencies on obtained ECCs can have on some TAMCNs. These 10 TAMCNs are 
items with a unit cost exceeding $ 1 million. 


Table 3-2. Sample of FSCs with ECCLSs and MSs with 90% Bootstrapped 
Confidence Intervals that Contain 0.0 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

Item 
Cost ($) 

B00157B 

2330 

Z BACKSCATTER RUGGEDIZED TRAILER (ZBRT) 

0 

0.0075 

72,080,000 

E08567K 

2350 

ASSAULT AMPHIBIOUS VEHICLE, RECOVERY 

0 

0.0486 

4,054,968 

E13787K 

2350 

RECOVERY VEHICLE, FT, HEAVY, W/EQUIP 

0 

0.0679 

2,400,000 

A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 

0.0125 

1,500,000 

A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 

0.0153 

1,500,000 

A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 

0.0153 

1,500,000 

A23067G 

2355 

GROUND SENSOR SURVEILLANCE VEHICLE (GSSV) 

0 

0.0465 

1,500,000 

A04997G 

5895 

DIGITAL TECHNICAL CONTROL (DTC) FACILITY 

0 

0.0253 

1,213,000 

A04997G 

5895 

DIGITAL TECHNICAL CONTROL (DTC) FACILITY 

0 

0.0379 

1,213,000 

A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 

0.0176 

1,000,000 


The same inaccuracy in the calculated ECC CARF values is observed for the rest of the 
140 ECC MSs and the 12 ECC LSs. Thus, the CARFs generated for a non-trivial 
number of TAMCNs are not significantly different from zero. Some of the CARFs 
affected include extremely high dollar value items, which means making allowancing 
decisions on ultimately unreliable CARF estimates could prove costly. 

2. Sensitivity Criteria Explained 

Because implications such as authorized allowance decisions and materiel 
acquisitions can be based on these CARF values, it is necessary to identify the range of 
unacceptable values of the ECC estimates. We measure what we call a sensitivity ratio to 
more effectively analyze the deviance from an acceptable range of values. Let the 90% 
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bootstrapped confidence interval for a particular TAMCN be given by [L,C/], where L 


represents the lower limit and U represents the upper limit. Then, the sensitivity ratio is 
given by: 


Sensitivity Ratio = 


( U~L \ 

< L J 


With this sensitivity ratio calculation, a resulting value of 1.0 or greater suggests 
the upper bound is twice the lower bound. We have deemed all CARFECCs with 
Sensitivity Ratios exceeding 1.0 as unacceptable. However, this subjective decision is 
ultimately up to the relevant decision maker, as such wide ranges may impact 
allowancing and budgeting decisions. 


a. ECCLA 

For the low level of conflict and in the assault phase, a total of 611 
ECCLAs have enough data to observe a sensitivity ratio analysis. Figure 3-1 shows the 
distribution of the sensitivity ratios for the ECC LAs. Of the 611 TAMCNs with 
ECC LAs, 82 (13%) have sensitivity ratios equal to or greater than 1.0. The maximum 
sensitivity ratio is 20.6. 


Sensitivity Criteria Ratio Results Distribution for ECC_LA 



Quantiles 

100.0% maximum 20.5978 
99.5% 15.1397 

97.5% 4.44096 

90.0% 1.55265 

75.0% quartile 0.3322 

50.0% median 0.13484 

25.0% quartile 0.03876 

10.0% 0.01795 

2.5% 0 

0.5% 0 

0.0% minimum 0 


Moments 

0.6105306 
1.7102032 
0.0691874 
0.746405 
0.4746561 
611 


Mean 
Std Dev 
Std Err Mean 
Upper 95% Mean 
Lower 95% Mean 
N 


Figure 3-1. Sensitivity Ratio Interval Distribution for ECC LA TAMCNs. 
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Only a sample of the top ECCLA TAMCNs, and the FSCs they fall 
under, are presented in Table 3-3, for those that reach the highest number greater than 1.0 
when analyzing the resulting sensitivity ratio. These high values of resulting sensitivity 
are meaningful in that they represent those ECC values that are extremely sensitive to 
minimal data variations. This makes the ECC values very unreliable, in terms of their 
employment in proper calculations for any sort of regression fits and analysis. The 
nomenclature is included in these tables to emphasize the diversity of equipment that can 
be affected. 


Table 3-3. Sample of the Top 10 of ECCLA CARF FSCs Failing Sensitivity Ratio 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

Sensitivity 

Criteria 

Ratio 

2355 

A23067G 

GROUND SENSOR SURVEILLANCE VEHICLE 

0.0005 

0.0102 

20.600 

2355 

A23067G 

SENSOR SYSTEM, MONITOR, MOBILE 

0.0005 

0.0093 

18.710 

5810 

A80447G 

LIMITED MAINT SPARE PARTS KIT 

0.0002 

0.0036 

15.310 

6930 

A70847G 

ANALYZER, NETWORK 

0.0002 

0.0024 

12.550 

8145 

C44332E 

CONTAINER, QUADRUPLE (QUADCON) 

le-5 

0.0001 

7.000 

6930 

A70847G 

LOCAL AREA NETWORK TEST SET 

0.0003 

0.0026 

6.900 

2350 

E00357K 

KIT, ARMOR, APPLIQUE 

0.0301 

0.2356 

6.826 

8145 

C44332E 

CONTAINER, QUADRUPLE (QUADCON) 

le-5 

0.0001 

6.450 

8145 

C44332E 

CONTAINER, QUADRUPLE (QUADCON) 

le-5 

0.0001 

6.450 

2590 

E09967M 

BLADE, MINE CLEARING 

0.0030 

0.0225 

6.422 


The severity of the impact of these highly sensitive ECCs, identified in 
Table 3-3, can be also emphasized by the TAMCNs’ cost. The highest-priced items are 
listed in Table 3-3. Topping this list is the case of TAMCN A23067G, Ground Sensor 
Surveillance Vehicle, at $1.5 million, together with its respective Mobile Sensor Monitor 
System with a cost of $657,000. 

b. ECCLS 

For the low level of conflict and in the sustainment phase, of a total of 519 
ECC_LSs, 303 (58%) have a sensitivity ratio equal to or greater than 1.0. This means that 
more than half of the available ECC LSs could be considered extremely sensitive to 
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small fluctuations in the data provided for its calculations. Figure 3-2 shows the 
distribution of sensitivity ratios. The values can reach as high as a maximum of 15.36. 


Sensitivity Criteria Ratio Results Distribution for ECC_LS 
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Quantiles 


Moments 


100.0% 

maximum 

15.3636 

Mean 

2.0098763 

99.5% 


12.9455 

Std Dev 

2.1990428 

97.5% 


8.45 

Std Err Mean 

0.0965273 

90.0% 


4.5 

Upper 95% Mean 

2.1995093 

75.0% 

quartile 

2.51648 

Lower 95% Mean 

1.8202432 

50.0% 

median 

1.2654 

N 

519 

25.0% 

quartile 

0.59052 



10.0% 


0.27202 



2.5% 


0.15549 



0.5% 


0 



0.0% 

minimum 

0 




Figure 3-2. Sensitivity Ratio Interval Distribution for ECCLS TAMCNs. 


The TAMCNs with the 10 highest sensitivity ratios are presented in 
Table 3-4. TAMCNs with high sensitivity ratios are extremely sensitive to minimal data 
variations. With the nomenclature included in the table, we can observe equipment that is 
critical to the operating forces and combat operations. 


Table 3-4. Sample of the Top 10 of ECC LS CARF FSCs Failing Sensitivity Ratio 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

Sensitivity 

Criteria 

Ratio 

3805 

B00637B 

TRACTOR, RUBBER TIRE, ARTICULATED 
STEERING, MP 

0.0002 

0.0034 

15.360 

6150 

B05797B 

DUMMY LOAD, GENERATOR SET, ELECT, 100KW, 
TRLR-MTD 

0.0003 

0.0042 

13.550 

6930 

A70847G 

ANALYZER, NETWORK 

0.0002 

0.0024 

12.550 

2320 

D10627K 

TRUCK, CARGO, 7 TON, XLWB, W/WINCH 

0.0003 

0.0037 

12.550 

5855 

E19477B 

TEST SET, NIGHT VISION 

0.0013 

0.0161 

11.550 

4930 

B11357B 

REFUELING SYSTEM, EXPEDIENT, HELO 

0.0016 

0.0186 

10.730 

2320 

D08867K 

TRUCK CARGO ARMOR 22.5 TON, 10X10, (LVSR) 

0.0004 

0.0047 

9.909 

2410 

B24607B 

TRACTOR, FT, W/ANGLE BLADE 

0.0002 

0.0024 

9.909 

6150 

B05797B 

DUMMY LOAD, GENERATOR SET, ELECT, 100KW, 
TRLR-MTD 

0.0005 

0.0052 

9.000 

2355 

E09467B 

LAV, COMMAND AND CONTROL (BN) 

0.0047 

0.0140 

2.000 
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Among the sample of identified highly sensitive ECCs in Table 3-4, the 
highest-value item is TAMCN E09467B, Command and Control LAV, with a cost of 
$3.25 million. 


c. ECC_MA 

For the medium level of conflict, in the assault phase, of a total of 611 
ECCMAs, 176 have a ratio equal to or greater than 1.0. Almost 30% of the available 
ECCMAs can be considered extremely sensitive to small fluctuations in the data 
provided for their calculations. 

Figure 3-3 shows the distribution of sensitivity ratios. The values can 
reach an extreme maximum of 98, which is indicative of a very unreliable ECC. 


Sensitivity Criteria Ratio Results Distribution for ECC_MA 
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Figure 3-3. Sensitivity Ratio Interval Distribution for ECC_MAs TAMCNs. 


A sample of the 10 TAMCNs with the highest sensitivity ratios are 
presented in Table 3-5. As for the ECC LAs and ECC LSs, TAMCNs with high 
sensitivity ratios are extremely sensitive to minimal data variations. 
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Table 3-5. Sample of the Top 10 of ECCMA CARF FSCs Failing Sensitivity Ratio 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

Sensitivity 

Criteria 

Ratio 

2355 

A23067G 

GROUND SENSOR SURVEILLANCE VEHICLE 

0.0005 

0.0469 

98.00 

2355 

A23067G 

SENSOR SYSTEM, MONITOR, MOBILE 

0.0005 

0.0427 

89.00 

5810 

A80447G 

LIMITED MAINT SPARE PARTS KIT 

0.0002 

0.0121 

53.81 

5855 

E19097B 

TEST SET, BORESIGHT COLLIMATOR 

0.0024 

0.0806 

33.10 

5820 

A00757G 

ARCHIVED TAMCN 

0.0001 

0.0026 

27.66 

1240 

E17802E 

SIGHT, BORE, MORTAR, W/CASE 

0.0003 

0.0063 

21.96 

5820 

A21717GL 

RADIO SET, VEHICULAR 

0.0006 

0.0124 

20.27 

1240 

E17802E 

SIGHT, BORE, MORTAR, W/CASE 

0.0003 

0.0056 

19.10 

5820 

A21717G 

RADIO SET, VEHICULAR 

0.0006 

0.0109 

17.63 

5820 

A21717GK 

RADIO SET, VEHICULAR 

0.0006 

0.0109 

17.63 


Among the samples of identified highly sensitive ECC_MAs in Table 3-5, 
the highest-value item is also TAMCN A23067G, Ground Sensor Surveillance Vehicle, 
with a cost of $ 1.5 million. 

d. ECCMS 

Finally, for the medium level of conflict and in the sustainment phase, of a 
total of 514 ECCMSs, 357 have a sensitivity ratio equal to or greater than 1.0. 
Figure 3-4 shows the distribution of sensitivity ratios. The values can reach a maximum 
of 36.2. 


Sensitivity Criteria Ratio Results Distribution for ECC_MS 



Quantiles 


Moments 


100.0% 

maximum 

36.165 

Mean 

2.6068909 

99.5% 


16.182 

Std Dev 

2.7813103 

97.5% 


8.74596 

Std Err Mean 

0.1226783 

90.0% 


5.75643 

Upper 95% Mean 

2.8479047 

75.0% 

quartile 

3.59364 

Lower 95% Mean 

2.3658772 

50.0% 

median 

1.88405 

N 

514 

25.0% 

quartile 

0.81114 



10.0% 


0.44747 



2.5% 


0.29168 



0.5% 


0.14417 



0.0% 

minimum 

0.12739 




Figure 3-4. Sensitivity Ratio Interval Distribution for ECC MSs TAMCNs. 
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The TAMCNs with the 10 highest sensitivity ratios for ECC_MSs are 
presented in Table 3-6. In addition, TAMCNs with high sensitivity ratios are extremely 
sensitive to minimal data variations. Their close resemblance to the TAMCNs depicted in 
Table 3-2 for the results of ECC LSs is evident. 


Table 3-6. Sample of the Top 10 of the ECC MS CARF FSCs Failing Sensitivity 

Ratio 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

Sensitivity 

Criteria 

Ratio 

2320 

D10627K 

TRUCK, CARGO, 7 TON, XLWB, W/WINCH 

0.0006 

0.0215 

36.170 

2320 

D10627K 

TRUCK, CHASSIS, XLWB, 7 TON, W/WINCH 

0.0011 

0.0185 

16.640 

6930 

A70847G 

ANALYZER, NETWORK 

0.0008 

0.0129 

15.850 

3805 

B00637B 

TRACTOR, RUBBER TIRE, ARTICULATED STEERING, MP 

0.0004 

0.0055 

14.680 

2320 

D10627K 

TRUCK, RTAA, CHASSIS, XLWB, 7 TON, W/WINCH 

0.0011 

0.0143 

12.630 

2410 

B24607B 

TRACTOR, FT, W/ANGLE BLADE 

0.0011 

0.0141 

11.590 

2430 

B24627B 

TRACTOR, FT, MEDIUM (CATERPILLAR) 

0.0026 

0.0302 

10.730 

2320 

D08867K 

TRUCK CARGO ARMOR 22.5 TON, 10X10, (LVSR) 

0.0005 

0.0050 

9.909 

2355 

E09467B 

LAV, COMMAND AND CONTROL (BN) 

0.0167 

0.0632 

2.780 

2355 

E09477M 

LAV, LIGHT ASSAULT, 25MM 

0.0065 

0.0179 

1.750 


Specifically, we can see TAMCN E09467B, the Command and Control 
FAV, as the highest-value item, with a cost of $3.25 million, followed by TAMCN 
E09477B, the Fight Assault FAV, with a cost of $3.22 million. 

While some of the FSCs represented in these tables refer to items of low- 
cost value, certain TAMCNs are essential combat assets of elevated monetary value in 
which an extreme sensitivity to minimal changes, identified in their ECC calculations, 
can have a hannful impact on the operating forces. Caution is also strongly emphasized 
since some of these high-dollar-value items affected by such small fluctuations in the 
available data may have elevated and widespread effects on the overall acquisition 
authorized allowances. 

Bootstrapping over the losses provides an example of a possible way to 
test sensitivity of calculations based on relatively limited usage data. By creating a test, 
or confidence interval we can provide a metric that can be used to compare the original 
historical data values presented and the predicted values obtained from methodology 
calculations. In this case, the results reveal that there is more sensitivity to small 
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fluctuations to data values in the sustainment phases than there is in the assault phases. 
This points to the observation that the existence of usage data does not necessarily ensure 
robust ECCs. 

B. ANALYZING COMPATABILITY OF CURRENT FSC AND FSG 

REGRESSION MODELS 

We were unable to replicate the Poisson regression models implemented 
in the legacy method for building FSC and FSG regression models. The sample of 
this regression model for an given FSC (FSG) is given by: 

Log e (CARF ECC_XX )=/) + /) (A AO _ TOTAL ) + {i (BEST _ COST)+ / 3 , (A AO _ TOTAL * BEST _ COST) 

n 

and is applied to every level and phase of conflict, i.e., ECCLA, ECCLS, etc. 

Because there are 155 FSCs, less than 49 FSGs, and as many as four regression 
models to build for each, constructing these models in JMP is highly inefficient and 
tedious. We import the data into SAS (see www.sas.com), build the regression models, 
and achieve a more efficient way to observe results. 

1, FSC Regression 

With the current FSC regression CARF (CRC) methodology we observe results 
that were inconsistent with a properly fitted model. We fit a GLM using a Poisson 
distribution and a log link function. Our response variable is ECC XX and the covariates 
are AAO TOTAL, BEST COST and AO_TOTAL*BEST_COST. For these, none of the 
Chi-Square tests are sufficient; most of the resulting F-statistics are greater than 0.9; and 
none of the p-values for any of the parameter estimates and their interactions are less than 
0.5, with most of them remaining above 0.7. These are symptoms of a poorly fitted model 
in which the distribution is wrongfully determined for its proper employment, rendering 
the resulting models invalid. 

We present an example of the GLM fit model results in an attempt to replicate the 
methodology suggested in the draft version of a contracted 2011 CARF study. This 
example is for FSC 2320, mostly rolling stock, i.e., MTVR 7-ton, refuelers, many 
HMMWV variants, etc., and comprising a total of 181 different TAMCNs. The results in 
this section are only for the response variable ECC LA and are presented in Figure 3-5. 
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The rest of the results for response variables ECCLS, ECCMA, and ECC MS can be 
found in Appendix D. A sample of the complete resulting files for the rest of the FSCs 
can be found in Appendix E (SAS Excel file). The complete electronic file can be made 
available on request. 


Generalized Linear Model Fit FSC=2320 

Response: ECC_LA 
Distribution: Poisson 
Link: Log 

Estimation Method: Maximum Likelihood 
Observations (or Sum Wgts) = 84 

Whole Model Test 

L-R 

Model -LogLikelihood ChiSquare DF Prob>ChiSq 

Difference 0.17655182 0.3531 3 0.9498 

Full 13.874747 

Reduced 14.0512988 


Goodness Of 

Fit Statistic ChiSquare 

Pearson 1.1301 

Deviance 1.1236 


DF Prob>ChiSq 

80 1.0000 

80 1.0000 


AlCc 

36.2558 

Effect Tests 


L-R 


Source DF ChiSquare Prob>ChiSq 

AAO_TOTAL 1 0.1814729 0.6701 

BEST_COST 1 0.0024058 0.9609 

AAO_TOTAL*BEST_COST 1 0.0449193 0.8322 


Parameter Estimates 


Term Estimate 

Intercept -2.90543 

AACLTOTAL -8.232e-6 

BEST_COST -2.375e-7 

(AAO TOTAL-21504.7)*(BEST_COST-201152) 5.207e-11 


L-R 


Std Error 

ChiSquare 

Prob>ChiSq 

Lower CL 

Upper CL 

1.1292597 

5.5141082 

0.0189* 

-5.55397 

-0.524541 

0.0000215 

0.1814729 

0.6701 

-8.253e-5 

2.225e-5 

4.8426e-6 

0.0024058 

0.9609 

-1.121e-5 

1.1766e-5 

2.438e-10 

0.0449193 

0.8322 

-3.49e-10 

3.865e-10 


Studentized Deviance Residual by Predicted 


ft ~ 

<D 

Q 


0.3 
0.2 
0.1 
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- 0.1 
- 0.2 
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Figure 3-5. GLM Fit for CARF FSC 2320 Y=ECC_LA. 
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From the JMP output presented in Figure 3-5, we observe less than optimal 
results, where the effects test and parameter estimates show p-values 0.6701 for 
AAOTOTAL, 0.9601 for BESTCOST, and 0.832 for AAO_TOTAL*BEST_COST. 
All remaining figures show the numeric significance predictors at similar and less than 
optimal results. 

Table 3-7 summarizes the results and presents the number of valid models 
obtained in the previously mentioned FSC CARF methodologies, where a valid model is 
one that has at least one statistically significant term with a p-value of less than 0.1. None 
of these responses are valid models. 


Table 3-7. Number of Valid Models in CARF FSC Regression (CRC) 


Model Described 

Number of Valid Models 

Total Number of Models with 

Generated from Total 

Data Available for Analysis 

Original CRC LA 

0 

58 

Original CRC LS 

0 

87 

Original CRC MA 

0 

58 

Original CRC MS 

0 

87 


2. FSG Regression 

Similar results are observed for the current FSG regression CARF (GRC) 
methodology, where inconsistency remains indicative of a poorly fitted model as a result 
of using a Poisson distribution, a GLM personality with a log link function, the 
AAO TOTAL, BEST COST, and the interaction AAO_TOTAL*BEST_COST as the 
influential tenns or numeric predictors. All Chi-Square tests are insufficient, the resulting 
F-statistics are greater than 0.9, and none of the p-values for any of the parameter 
estimates and their interactions show less than 0.5, with most of them remaining well 
above 0.7. These, as well, are symptoms of improperly fitted and invalid models. 

An example of the GLM fit model results for FSG 12, Fire Control Equipment, 
which comprises a total of 22 different TAMCNs, of which only seven have ECCs, is 
presented in Figure 3-6 and, as in the previous example, only the results for response 
variable ECC LA are presented. The rest can be found in Appendix F for response 
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variable ECC LS, ECCMA and ECCMS. A sample of the complete resulting files for 
the rest of the FSGs can also be found in Appendix G (SAS Excel file). The complete 
electronic file can be made available on request. 


Generalized Linear Model Fit FSG=12 

Response: ECC_LA 
Distribution: Poisson 
Link: Log 

Estimation Method: Maximum Likelihood 
Observations (or Sum Wgts) = 7 

Whole Model Test 

L-R 


Model -LogLikelihood 

Difference 0.00318211 

Full 0.09718339 

Reduced 0.1003655 


ChiSquare 

0.0064 


DF Prob>ChiSq 

3 0.9999 


Goodness Of 

Fit Statistic ChiSquare 

Pearson 0.0019 

Deviance 0.0020 


DF Prob>ChiSq 

3 1.0000 

3 1.0000 


AlCc 

28.1944 

Effect Tests 

L-R 


Source 

AAO_TOTAL 

BESTCOST 

AAO_TOTAL”BEST_COST 


DF ChiSquare Prob>ChiSq 

1 0.0005542 0.9812 

1 0.0013269 0.9709 

1 0.0015711 0.9684 


Parameter Estimates 


L-R 

Std Error ChiSquare Prob>ChiSq Lower CL Upper CL 


Term Estimate 

Intercept -4.359791 

AACLTOTAL -0.000086 

BEST_COST 0.0006023 

(AAO_TOTAL-24338.9)*(BEST_COST-6889.45) -2.757e-8 


72.278841 

0.0041525 

0.9486 

-1486.12 

886.10571 

0.0035757 

0.0005542 

0.9812 

-0.056701 

0.0642424 

0.016238 

0.0013269 

0.9709 

-0.259278 

0.2757727 

6.8369e-7 

0.0015711 

0.9684 

-1.111 e-5 

1.0654e-5 


Studentized Deviance Residual by Predicted 


0.04 
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Figure 3-6. GLM Fit for CARF FSG 12 with Y=ECC_FA. 
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From the JMP output presented in Figure 3-6, we can observe the same less than 
optimal results in the FSG 12 GLM regression. Here, the effects test and parameter 
estimates show p-values above 0.95 for all numeric predictor tenns. Similar invalid 
results are observed for the rest of the GLM regressions for this FSG, and presented in 
the appropriate appendix. 

Table 3-8 is a summary of the number of valid models obtained, where a valid 
model is one that has at least one term significance with a p-value of less than 0.1, when 
performing the fitting of the model for the proposed CARF methodology for calculating 
GRCs. As the same results obtained for FSC 2320, none of these models are valid. 


Table 3-8. Number of Valid Models in CARF FSG Regression (GRC) 


Model Described 

Number of Valid Models 

Total Number of Models with 

Generated from Total 

Data Available for Analysis 

Original GRC LA 

0 

28 

Original GRC LS 

0 

36 

Original GRC MA 

0 

28 

Original GRC MS 

0 

36 


The results presented in this section are the results obtained in performing the 
research for this thesis. While a full effort was made to replicate the exact procedure 
employed in the source code provided in the JMP file, the same exact resulting values of 
the calculations were not obtained. The obtained values point to inconsistent models with 
less desirable regression fits and where no models show validity. The effects of invalid 
regression models observed in obtaining CRC values are further noticed when obtaining 
GRC values. 

C. ORDINARY LEAST SQUARES (OLS) FSC AND FSG REGRESSION 

In this section, because we were unable to recreate the CRCs and GRCs using 
GLM, we construct corresponding regression models using Ordinary Least Squares. The 
results obtained under this approach are a major improvement in terms of model 
performance and number of CRCs and GRCs generated. The F-statistics obtained and the 
overall p-values observed, on some of the CRC and GRC regressions, are of a more 
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acceptable statistical significance for the numeric predictors used. When the OLS 
regression fits are subjected to the same threshold as the GLM regressions, an F-statistic 
lower than 0.1 and at least one of the regression tenns with a p-value of less than 0.1, we 
observe that, in some cases these fits become valid for the different levels and phases of 
conflict. We also observe that some of these better-behaved models make a more efficient 
use of the sparse historical data available, although such is not always the case. 

1. OLS FSC Regression 

For the OLS FSC regression, we use the same example of FSC 2320 as is used for 
the GLM regressed FSC CARF (CRC) methodology. Better-behaved models, showing 
consistency in the numeric predictors used and statistical significance in the terms’ p- 
values, are observed. The results of this are presented in Table 3-9 for FSC ECCLA, 
Table 3-10 for FSC ECCLS, Table 3-11 for FSC ECC MA, and Table 3-12 for FSC 
ECC MS. Each table shows the resulting p-values for each of the terms used in the OLS 
regression and the estimate value, the standard error resulted, and the t value, for each 
coefficient. All of these OLS regression samples for FSC 2320 result in valid models. 
Although the resulting p-values for the interaction tenn AAO_TOTAL*BEST_COST 
remain above the significance threshold of 0.1, the rest of the terms for the OLS 
regressed example FSC 2320 remain at significant p-values well below 0.1. 


Table 3-9. OLS fit for CARF FSC 2320 for Y=ECC LA 


OLS REGRESSED CARF FSC LA 



FSC 2320 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.065 

0.005 

12.28 

4.13e-20 

AAO TOTAL 

-5.61e-7 

2.17e-7 

-2.58 

0.0115 

BEST COST 

-6.39e-8 

2.43e-8 

-2.62 

0.0102 

AAO TOTAL*BEST COST 

1.53e-12 

l.lle-12 

1.37 

0.1734 


Table 3-10. OLS fit for CARF FSC 2320 for Y=ECC LS 


OLS REGRESSED CARF FSC LS 



FSC 2320 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.006 

0.0008 

7.51 

7.20e-ll 

AAO TOTAL 

-7.08e-8 

3.37e-8 

-2.09 

0.0390 

BEST COST 

-1.06e-8 

3.77e-9 

-2.81 

0.0061 

AAO TOTAL*BEST COST 

2.47e-13 

1.73e-13 

1.43 

0.1560 
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Table 3-11. OLS fit for CARF FSC 2320 for Y=ECC MA 


OLS REGRESSED CARF FSC MA 



FSC 2320 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.074 

0.005 

12.87 

3.39e-21 

AAO TOTAL 

-6.34e-7 

2.37e-7 

-2.66 

0.0092 

BEST COST 

-7.94e-8 

2.66e-8 

-2.98 

0.0037 

AAO TOTAL*BEST COST 

1.65e-12 

1.22e-12 

1.35 

0.1790 


Table 3-12. OLS fit for CARF FSC 2320 for Y=ECC MS 


OLS REGRESSED CARF FSC MS 



FSC 2320 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.015 

0.001 

8.69 

3.5 le-13 

AAO TOTAL 

-1.43e-7 

7.44e-8 

-1.93 

0.0568 

BEST COST 

-2.62e-8 

8.32e-9 

-3.14 

0.0023 

AAO TOTAL*BEST COST 

3.71e-13 

3.81 e-13 

0.97 

0.3335 


As a result of the OLS modeling methodology, we obtain more valid models 
when compared to the proposed methodology approach. Table 3-13 is a summary of the 
number of valid models obtained, where, once again, a valid model is one that has at least 
one term significance with a p-value of less than 0.1, when performing the fitting of the 
model using a OLS regression method for calculating CARF FSCs. 


Table 3-13. Number of Valid Models in OLS CRC CARF Regression 


Model Described 

Total Number of 
Models with Data 
Available for 
Analysis 

Number of Valid 
Models Generated 
from Total 

Total Number of 
TAMCN (CEC 1) 
CRCs Generated 
by OLS 

Number of CARF 
CRCs Generated 
by GLM 

OLS CRC LA 

58 

13 

270 

190 

OLS CRC LS 

87 

16 

496 

197 

OLS CRC MA 

58 

14 

369 

2 

OLS CRC MS 

87 

16 

586 

3 


In this table, the “Total Number of Models with Data Available for Analysis” 
column, expressed by the applicable CRC level and phase of conflict, represents the 
overall number of models observed to have all the data necessary to perfonn the analysis. 
The “Number of Valid Models Generated from Total” refers to the results of the validity 
comparison for term significance from the previous column totals. The “Total number of 
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TAMCN (CEC 1) CRCs Generated” provides the total possible number of TAMCNs that 
have been assigned a CRC value, only for CEC 1, and only for that level and phase of 
conflict. The last column is provided for the purpose of comparison to the original totals 
obtained from the file provided for the study of this thesis as part of a draft version of a 
originally proposed CARF methodology and refers to the total CARF values assigned. 
This same description applies to the rest of these similar tables presented in this section. 
OLS produces a substantially higher number of CRCs than does GLM, especially for MA 
and MS levels. 

2. OLS FSG Regression 

Better-behaved models showing consistency in the numeric predictors used and 
statistical significance in the terms’ p-values are observed when we follow an 
OLS-regressed FSG CARF methodology (GRC) for the same example of FSG 12, as is 
used in the GLM-regressed GRC methodology. The results of these are presented in 
Tables 3-14 for FSG ECC LA, 3-15 for FSG ECCLS, 3-16 for FSG ECC MA, and 
3-17 for FSG ECC_MS. Each table shows the resulting p-values, the estimate value, the 
standard error resulted, and the t value for each coefficient. All the models for FSG 12 are 
valid, showing an improvement for term significance p-values, with most of them 
remaining below 0.1. A few exceptions are observed, as is the case of the value for the 
AAO TOTAL and the interaction term, when the response variable is ECC LS and 
ECC_MS. Nevertheless, the rest of the regressed model fits remain valid based on our 
previously established validity threshold. 


Table 3-14. OLS CARF FSG 12 fit for Y=ECC LA 


OLS REGRESSED CARF FSG LA 



FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

-.0003 

0.0009 

-0.430 

0.6891 

AAO TOTAL 

1.37e-7 

5.16e-8 

2.65 

0.0565 

BEST COST 

1.48e-7 

5.98e-8 

2.48 

0.0679 

AAO TOTAL*BEST COST 

-5.67e-12 

2.54e-12 

-2.23 

0.0893 
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Table 3-15. OLS CARF FSG 12 fit for Y=ECC LS 


OLS REGRESSED CARF FSG LS 



FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.0003 

0.0001 

3.21 

0.0092 

AAO TOTAL 

-,43e-12 

4.34e-10 

-0.01 

0.9848 

BEST COST 

7.26e-8 

1.65e-8 

4.39 

0.0013 

AAO TOTAL*BEST COST 

2.32e-13 

4.24e-13 

0.54 

0.5965 


Table 3-16. OLS CARF FSG 12 fit for Y=ECC MA 


OLS REGRESSED CARF FSG MA 



FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.00008 

0.0007 

0.107 

0.9193 

AAO TOTAL 

1.70e-7 

4.31e-8 

3.95 

0.0167 

BEST COST 

4.19e-7 

4.99e-8 

8.39 

0.0011 

AAO TOTAL*BEST COST 

-,14e-12 

2.12e-12 

-2.89 

0.0443 


Table 3-17. OLS CARF FSG 12 fit for Y=ECC MS 


OLS REGRESSED CARF FSG MS 



FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.001 

0.0003 

3.25 

0.0086 

AAO TOTAL 

-,80e-10 

1.24e-9 

-0.70 

0.4951 

BEST COST 

3.36e-7 

4.73e-8 

7.10 

0.00003 

AAO TOTAL*BEST COST 

1.16e-12 

1.21e-12 

0.95 

0.3602 


Table 3-18 is a summary of the number of valid models obtained, where a valid 
model is one that has at least one term significance with a p-value of less than 0.1, when 
performing the fit using OLS methodology for GRCs. 


Table 3-18. Number of Valid Models in OLS GRC CARF Regression 


Model Described 

Total Number of 
Models with Data 
Available for 
Analysis 

Number of Valid 
Models Generated 
from Total 

Total Number of 
TAMCN (CEC 1) 
GRCs Generated 
by OLS 

Number of 
CARF GRCs 
Generated by 
GLM 

OLS GRC LA 

28 

11 

398 

46 

OLS GRC LS 

36 

11 

406 

51 

OLS GRC MA 

28 

11 

707 

59 

OLS GRC MS 

36 

10 

605 

65 
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3. 


OLS Aggregate FSG Regression (by ECC and FSC) 


Though the draft version of CARF methodology mentions that every GRC 
regression model is built on the results of the ECCs obtained and the CRC regressions, 
we are unable to identify specific traces that could show this approach was performed. 
With that in mind, we test the result that can be obtained by following the specific 
assignment algorithm in the decision flow map, presented in Figure 1-1, for a proposed 
FY2011 CARF assignment methodology. In doing so, the previously calculated ECCs 
and the obtained FSC CARFs (CRCs), for all levels and phases of conflict, are provided 
as inputs to the constructed model used to obtain FSG CARFs (GRCs). The results of 
such an approach under an OLS methodology, using the previously employed example of 
FSG 12, are presented in Tables 3-19 for CARF FSG LA, 3-20 for CARF FSG LS, 3-21 
for CARF FSG MA, and 3-22 for CARF FSG MS. In addition, each table shows the 
resulting p-values, the estimate value, the standard error resulted, and the t value for each 
coefficient. With the exception of the regressed fit when the response variables of 
ECC LS and ECC MS are used, the rest are very stable and valid models. Even with the 
explicit exceptions, at least one term—in both cases, the interaction term—remains well 
below a p-value of 0.1, making the overall regression fit good, thus suggesting a valid 
model. 


Table 3-19. OLS CARF FSG 12 LA (created by ECCs and FSC inputs) 


OLS REGRESSED CARF FSG LA (Using ECCs and FSCs) 

FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

-.038 

0.0166 

-2.29 

0.1050 

AAO TOTAL 

0.00006 

0.00001 

5.06 

0.0148 

BEST COST 

2.02e-7 

3.83e-8 

5.28 

0.0132 

AAO TOTAL*BEST COST 

-7.62e-10 

1.33e-10 

-5.70 

0.0106 


Table 3-20. OLS CARF FSG 12 LS (created by ECCs and FSC inputs) 


OLS REGRESSED CARF FSG LS (Using ECCs and FSCs) 

FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.004 

0.0006 

6.17 

0.0034 

AAO TOTAL 

-,32e-9 

4.73e-9 

-0.49 

0.6494 

BEST COST 

1.Ole-9 

6.70e-10 

1.51 

0.2050 

AAO TOTAL*BEST COST 

-8.33e-12 

1.93e-12 

-4.31 

0.0125 
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Table 3-21. OLS CARF FSG 12 MA (created by ECCs and FSC inputs) 


OLS REGRESSED CARF FSG MA (Using ECCs and FSCs) FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

-.04 

0.018 

-2.14 

0.1210 

AAO TOTAL 

0.00007 

0.00001 

4.95 

0.0157 

BEST COST 

2.23e-7 

4.31e-8 

5.18 

0.0139 

AAO TOTAL*BEST COST 

-8.44e-10 

1.50e-10 

-5.61 

0.0111 


Table 3-22. OLS CARF FSG 12 MS (created by ECCs and FSC inputs) 


OLS REGRESSED CARF FSG MS (Using ECCs and FSCs) 

FSG 12 

Coefficients 

Estimate 

Std. Error 

t value 

Pr( > 111) 

Intercept 

0.01 

0.001 

5.53 

0.0052 

AAO TOTAL 

-,94e-9 

1.34e-8 

-0.29 

0.7831 

BEST COST 

2.45e-9 

1.89e-9 

1.29 

0.2652 

AAO TOTAL*BEST COST 

-2.1le-11 

5.47e-12 

-3.85 

0.0182 


Table 3-23 is a summary of the number of valid models obtained when 
aggregating ECCs and CRCs to calculate GRCs, identified in the table as OLS 
AGGREGATE GRC XX, where a valid model is one that has at least one term 
significance with a p-value of less than 0.1. 


Table 3-23. Number of Valid Models in CARF FSG OLS Regression (created with 

ECC and CRC input) Methodology 


Model Described 

Total Number of 
Models with Data 
Available for 
Analysis 

Number of 
Valid Models 
Generated 
from Total 

Total Number of 
TAMCN (CEC 1) 
GRCs Generated 

Number of 
CARF GRCs 
Assigned by 
GLM 

OLS AGGREGATED GRC LA 

55 

16 

540 

46 

OLS AGGREGATED GRC LS 

60 

14 

414 

51 

OLS AGGREGATED GRC MA 

55 

12 

308 

59 

OLS AGGREGATED GRC MS 

60 

16 

708 

65 


Looking at the number of resulting models of the aggregated OLS GRC 
methodology, we see an increase obtained from making use of the available calculated 
values of ECCs and CRCs. Both the number of models with data available for analysis, 
followed by the number of valid models obtained, provide the resulting increase in 
overall TAMCNs able to have a calculated CARF CRC model. The last column in 
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Tables 3-22 and 3-23 present the results of the originally reported GRC models from 
Table 2-3. The increase, obtained when following an OLS and an OLS aggregated 
approach in usable models, is indicative of an improvement and of another possible 
approach in which further research could be invested. It is also prudent to emphasize here 
that the results presented for OLS regression have been of an exploratory nature and have 
not been fully validated by this thesis. Furthermore, no other analytical parameters than 
the ones mentioned here were specifically traced or studied with the intention to validate 
the before-mentioned procedure. Nevertheless, in finding the results depicted in this 
section, the need for further research into more robust and less labor-intensive practices 
and methodologies has become evident. 

D. CROSS-VALIDATION APPROACH TO IDENTIFY DATA 

SUFFICIENCY FOR EFFECTIVE ANALYSIS 

In this section, we cross validate a small sample of our regression models. The 
primary intent is to evaluate the performance of the models, but we also hope to gain 
some insight in detennining how sensitive the models are to gaps in the data. 

In every trial, we withhold different percentages of the available TAMCNs data. 
With this, we can run the regression models and compare the accuracy of that individual 
model to produce resulting values similar to those from the originally regressed values of 
the data withheld. In every case, the percentages of data withheld are 20%, 15%, and 
10%. The examples we use are FSCs 2320 and 5820 and FSG 23. These FSCs and FSG 
are personally chosen because of their amount of available CEC 1 TAMCNs with 
available data and because of their primary importance, as rolling stock and 
communications equipment, to the operating forces. 

Every example that follows employs the OLS-regressed procedure and presents 
the resulting summary of fit of that particular model, in every level and phase of conflict, 
having the specific percentage of data withheld and the explanation of the total numbers 
used. Also presented are the observed results’ distributions for that specific percentage of 
data withheld and the resulting predicted values. A table at the end of every example 
compiles these resulting numbers and identifies which ones remained inside the 
90-percentile confidence interval, which was explained in Section A of this chapter. For 
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the FSG example, only results for conflict level and phases LA and MS are presented 
with the intention of just emphasizing the most relevant results. 

1. Example Using FSC 2320, Wheeled Trucks and Truck Tractors 

Using the previous example of FSC 2320, where a total of 181 TAMCNs are 
available and we have originally 84 ECC_LAs, ECC_LSs, ECC_MAs, and ECC_MSs, 
we perfonn a cross validation in order to identify symptoms the regression would show 
when randomly withholding 20% (17 of the 84 observations withheld) of the available 
data. Figure 3-7 shows the resulting values for the OLS regression fit for ECC LA. 


Response ECC_LA FSC=2320 
Summary of Fit 

RSquare 
RSquare Adj 
Root Mean Square Error 
Mean of Response 
Observations (or Sum Wgts) 


Analysis of Variance 


Source 

DF 

Sum of 
Squares 

Mean Square 

F Ratio 

Model 

3 

0.00735324 

0.002451 

4.4789 

Error 

63 

0.03447662 

0.000547 

Prob > F 

C. Total 

66 

0.04182986 


0.0065* 


Parameter Estimates 


Term 

Estimate 

Std Error 

t Ratio 

Prob>ltl 

Intercept 

0.0489275 

0.007166 

6.83 

<.0001* 

AAO„TOTAL 

-1,758e-7 

8.455e-8 

-2.08 

0.0416* 

BEST_COST 

-8.666e-9 

2.526e-8 

-0.34 

0.7327 

(AAO_TOTAL-26534.1)*(BEST_COST-221217) 

1.476e-12 

1.06e-12 

1.39 

0.1688 


Figure 3-7. OLS Fit for CARF FSC 2320 with Y=ECC_LA When 20% of TAMCN 

Data is Withheld. 


0.175789 

0.136541 

0.023393 

0.042482 

67 


In this example, there is only one numeric predictor term with a p-value of less 
than 0.1 and one that closely approaches that threshold. These are sufficient enough 
results to observe this regression fit as valid based on our previously established model 
validity criteria of at least one term’s p-value being less than 0.1. 
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The distribution of the withheld data for comparison of the results of the cross 
validation procedure is presented in Figure 3-8. The distribution of the original ECC LAs 
shows a wide range (0.0032, 0.1077), while the distribution for the predicted CRC_LAs, 
with 20% data randomly withheld, shows a very much-restricted range, (0.0404, 0.0626). 
This, in turn, translates not into predicted values remaining inside the wide range of 
original values, but actually failing to remain inside the previously calculated confidence 
interval, as presented in Table 3-24. 


Distributions 
ECC_LA Original 



Quantiles 


Predicted CRC LA 



100 . 0 % 

99.5% 

97.5% 

90.0% 

75.0% 

50.0% 

25.0% 

10 . 0 % 

2.5% 

0.5% 

0 . 0 % 


maximum 0.10775 
0.10775 
0.10775 
0.1055 
quartile 0.05632 
median 0.04288 
quartile 0.03456 
0.01505 
0.00328 
0.00328 
minimum 0.00328 


Quantiles 


100 . 0 % 

99.5% 

97.5% 

90.0% 

75.0% 

50.0% 

25.0% 

10 . 0 % 

2.5% 

0.5% 

0 . 0 % 


Moments 

Mean 
Std Dev 
Std Err Mean 
Upper 95% Mean 
Lower 95% Mean 
N 


maximum 0.06262 
0.06262 
0.06262 
0.06128 
quartile 0.05691 
median 0.05115 
quartile 0.04179 
0.04045 
0.04045 
0.04045 
minimum 0.04045 


0.0490712 

0.0264895 

0.0064246 

0.0626909 

0.0354516 

17 


Moments 

Mean 
Std Dev 
Std Err Mean 
Upper 95% Mean 
Lower 95% Mean 
N 


0.0496847 

0.0079132 

0.0019192 

0.0537533 

0.0456161 

17 


Figure 3-8. Comparison Distributions of Original ECC LA and Predicted CRC LA 

for the 20% Withheld. 
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Table 3-24. Predicted CRCLA and Original ECCLA comparison with Cl, for the 

20% withheld 


TAMCN 

ECC_LA 

Original 

Predicted 

CRCLA 

Interval 
Lower Bound 

Interval 
Upper Bound 

Predicted CRC_LA in Cl 

D00337KA 

0.0033 

0.0543 

0.0031 

0.0037 

No 

D01987K 

0.0429 

0.0404 

0.0420 

0.0439 

No 

D01987K 

0.0429 

0.0404 

0.0421 

0.0437 

No 

D01987KA 

0.0429 

0.0512 

0.0422 

0.0438 

No 

D02097K 

0.1049 

0.0416 

0.1040 

0.1058 

No 

D08777K 

0.0314 

0.0512 

0.0292 

0.0373 

No 

D08817K 

0.0569 

0.0530 

0.0550 

0.0597 

No 

D08867K 

0.0180 

0.0419 

0.0161 

0.0204 

No 

D10627K 

0.0377 

0.0469 

0.0362 

0.0397 

No 

D10627K 

0.0377 

0.0426 

0.0362 

0.0397 

No 

D10727K 

0.1078 

0.0527 

0.1016 

0.1144 

No 

D10737K 

0.0289 

0.0440 

0.0282 

0.0302 

No 

D11257K 

0.0481 

0.0626 

0.0452 

0.0508 

No 

D11587KA 

0.0678 

0.0609 

0.0671 

0.0683 

No 

D11597K 

0.0558 

0.0601 

0.0556 

0.0560 

No 

D11597K 

0.0558 

0.0595 

0.0556 

0.0560 

No 

D12137K 

0.0516 

0.0412 

0.0493 

0.0541 

No 


In Table 3-24, we can observe the obtained predicted CRC values for the LA, 
only for the 20% data randomly withheld, as compared to the originally obtained ECC 
values. The lower and upper interval columns are from the calculated confidence 
intervals obtained and presented in Section A of this chapter. The last column of this 
table confirms if the obtained predicted value falls inside that confidence interval. In this 
case, there are no predicted CRC LAs that meet this criteria. For FSC 2320 with an LA 
level and phase of conflict, even missing just 20% of TAMCNs with available data (i.e., 
17 of 84) renders the regressed CRC values unreliable. 

The rest of these figures and tables have been omitted here, but can be found in 
Appendix H. The compiled results have been properly tabulated and are discussed at the 
end of this section. 

When we withhold roughly 15% (12 of the 84 TAMCNs) and 10% (8 of the 84 
TAMCNs) of the available data for the same FSC 2320, the resulting summary of fits for 
ECC_LA show valid regressions. Nevertheless, even when only withholding 10% of the 
data, the results do not improve. None of the obtained predicted CRC values in both cases 
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remain inside the determined confidence interval. For this level and phase of conflict, the 
regressed CRC values are likely marginally reliable. 

Following the same cross-validation approach on FSC 2320, with a 20% of 
TAMCN data randomly withheld, for ECC_LS we can observe that the regression fit 
remains valid when we consider that the cross-effect tenn has a resulting p-value of less 
than 0.1, as shown in Figure 3-9. 
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Response ECC_LS FSC=2320 


Summary of Fit 

RSquare 0.082866 

RSquare Adj 0.039193 

Root Mean Square Error 0.002742 

Mean of Response 0.002633 

Observations (or Sum Wgts) 67 


Analysis of Variance 


Source 

DF 

Sum of 
Squares 

Mean Square 

F Ratio 

Model 

3 

0.00004281 

0.000014 

1.8974 

Error 

63 

0.00047383 

7.521 e-6 

Prob > F 

C. Total 

66 

0.00051664 


0.1391 


Parameter Estimates 

Term 

Intercept 

AAO_TOTAL 

BEST_COST 

(AAO_TOTAL-26534.1)*(BEST 


Estimate 

0.0026331 
-2.818e-9 
2.477e-10 
COST-221217) 2.16e-13 


Std Error 

t Ratio 

Prob>ltl 

0.00084 

3.13 

0.0026* 

9.912e-9 

-0.28 

0.7771 

2.961 e-9 

0.08 

0.9336 

1.24e-13 

1.74 

0.0871 


Figure 3-9. OLS Fit for CARF FSC 2320 with Y=ECC_LS When 20% of TAMCN 

Data is Withheld. 


Distributions 
ECC_LS Original 





1—1 

0 o.c 

05 0.01 0.015 


Predicted CRC LS 


h$h 

i 



0 

0.0 

05 0.01 0.015 


Quantiles 


Moments 


100.0% 

maximum 

0.0158 

Mean 

0.0041095 

99.5% 


0.0158 

Std Dev 

0.003959 

97.5% 


0.0158 

Std Err Mean 

0.0009602 

90.0% 


0.00976 

Upper 95% Mean 

0.0061451 

75.0% 

quartile 

0.00592 

Lower 95% Mean 

0.002074 

50.0% 

median 

0.00353 

N 

17 

25.0% 

quartile 

0.00134 



10.0% 


0.00016 



2.5% 


0.00016 



0.5% 


0.00016 



0.0% 

minimum 

0.00016 




Quantiles 


Moments 


100.0% 

maximum 

0.00585 

Mean 

0.0039051 

99.5% 


0.00585 

Std Dev 

0.0011164 

97.5% 


0.00585 

Std Err Mean 

0.0002708 

90.0% 


0.00562 

Upper 95% Mean 

0.0044791 

75.0% 

quartile 

0.00486 

Lower 95% Mean 

0.003331 

50.0% 

median 

0.00386 

N 

17 

25.0% 

quartile 

0.00307 



10.0% 


0.00254 



2.5% 


0.00213 



0 . 5 % 


0.00213 



0.0% 

minimum 

0.00213 




Figure 3-10. Comparison of Distributions of Original ECCLS and Predicted CRCLS 

for the 20% Withheld. 
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The distributions presented in Figure 3-10 for the original ECCLSs and the 
predicted CRCLSs show similar results than those of the distributions presented for 
ECCLA and the predicted CRCLA, but for this level and phase of conflict, when we 
compare the resulting predicted CRC values to the previously established confidence 
intervals, we are able to identify that seven of the 17 TAMCNs (Table 3-25) with data 
that were randomly selected and withheld were predicted values that remained inside the 
mentioned interval. This suggests that, in this level and phase of conflict, and possibly 
because the available data is less sparse here, we are still able to consider valuable the 
reliance on some of the predicted values. 


Table 3-25. Predicted CRCLS and Original ECC LS Comparison with Cl, for the 

20% Withheld 


TAMCN 

ECC_LS 

Original 

Predicted 

CRCLS 

Interval 
Lower Bound 

Interval 
Upper Bound 

Predicted CRC_LS in Cl 

D00337KA 

0.0002 

0.0044 

0.0001 

0.0006 

No 

D01987K 

0.0035 

0.0032 

0.0027 

0.0046 

Yes 

D01987K 

0.0035 

0.0032 

0.0027 

0.0044 

Yes 

D01987KA 

0.0035 

0.0039 

0.0028 

0.0044 

Yes 

D02097K 

0.0044 

0.0031 

0.0035 

0.0052 

No 

D08777K 

0.0082 

0.0039 

0.0060 

0.0142 

No 

D08817K 

0.0079 

0.0044 

0.0061 

0.0108 

No 

D08867K 

0.0024 

0.0030 

0.0004 

0.0047 

Yes 

D10627K 

0.0020 

0.0034 

0.0005 

0.0040 

Yes 

D10627K 

0.0020 

0.0026 

0.0005 

0.0040 

Yes 

D10727K 

0.0158 

0.0041 

0.0097 

0.0225 

No 

D10737K 

0.0007 

0.0028 

0.0006 

0.0020 

No 

D11257K 

0.0074 

0.0059 

0.0045 

0.0102 

Yes 

D11587KA 

0.0042 

0.0056 

0.0035 

0.0046 

No 

D11597K 

0.0002 

0.0054 

0.0001 

0.0004 

No 

D11597K 

0.0002 

0.0053 

0.0001 

0.0004 

No 

D12137K 

0.0037 

0.0021 

0.0014 

0.0062 

No 


The rest of these figures and tables have been omitted here, but can be found in 
Appendix H. The compiled results have been properly tabulated and are discussed at the 
end of this section. 

When randomly withholding 15% and 10% of the data on FSC 2320 for ECC_LS, 
the regression fits also remain valid. We also observe that as we withhold less data, in 
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this case and for this level and phase of conflict, the numbers of predicted CRC values 
that remain inside the confidence interval improve. This suggests that most of the 
predicted CRC values are still reliable. 

Similar results are observed when randomly withholding the same depicted 
amounts of data throughout the rest of this analysis for ECCMAs and ECCMSs. The 
rest of the results for FSC 2320, at every level and phase of conflict and with the different 
amount of TAMCN data withheld, are compiled in Table 3-26. Though we have 
established a flexible 90% confidence interval, only one model, CRC LS, is able to show 
predicted values inside this interval at a mere 75%. As well, it is evident that CRC_LA 
shows no acceptable performance and does not maintain any predicted values inside the 
bootstrapped confidence interval developed. 


Table 3-26. Cross Validation Compiled Results for FSC 2320 


FSC 

2320 

20% Data Withheld 
(17 of 84) 

15% Data Withheld 
(12 of 84) 

10% Data Withheld 
(8 of 84) 

Number of CRCs 
in Cl 

Percent in 
Cl 

Number of CRCs 
in Cl 

Percent in 
Cl 

Number of CRCs 
in Cl 

Percent in 
Cl 

LA 

Oof 17 

0% 

Oof 12 

0% 

1 of 9 

0% 

LS 

7 of 17 

41% 

6 of 12 

50% 

6 of 9 

75% 

MA 

3 of 17 

18% 

2 of 12 

17% 

3 of 9 

25% 

MS 

8 of 17 

47% 

6 of 12 

50% 

6 of 9 

50% 


When comparing the overall results of the cross-validation for FSC 2320, we can 
observe that the assault phases for both levels of conflict are the ones with the least 
resiliency and least reliability for predicted CRC values. As far as the low and medium 
levels, in the sustainment phases, we see that even when we withhold 10% of the data the 
model has trouble generating estimates that are inside the bootstrapped CIs. 
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2. Example Using FSG 23, Motorized Vehicles 

For the cross-validation approach on an FSG, we used FSG 23, Motor Vehicles, 
Trailers, and Cycles, with a total of 324 TAMCNs CEC 1 available and broken down in 
FSCs, as shown in Figure 3-11. 



FSC 

FSG 

2310 

2320 

2330 

2340 

2350 

2355 

23 

7 

181 

60 

4 

16 

56 


Figure 3-11. Number of TAMCNs in FSG 23 by FSC. 

From the original data, of these 324 TAMCNs, 180 each have ECC_LAs, 
ECC LSs, ECC MAs, and ECCMSs. 

The resulting figures and tables for this cross validation have also been omitted 
here, but can be found in Appendix I. The compiled results have been properly tabulated 
and are discussed at the end of this section. 

The effects observed when randomly withholding 20% (36 of the 180 
observations withheld) of the available data and performing an OLS regression with the 
response variable as ECCLA, show that the regression fit remains valid, but when we 
compare the resulting predicted CRC values to the previously established confidence 
intervals, only 2 of the 36 TAMCNs remained inside the mentioned interval. This 
suggests that, in this level and phase of conflict, and for this FSG, only some of these 
regressed predicted values could be considered reliable. 

The same results are observed when randomly withholding only 15% (27 of 180) 
of the data for the ECC LA. The regression fit and none of the resulting predicted CRC 
values remain inside the mentioned interval. In withholding only 10% (18 of 180) of the 
data for ECC LA, the regression lit remains valid with only 2 of the 18 predicted CRC 
values remaining inside the mentioned interval. 

The rest of the results for FSG 23, at every level and phase of conflict and with 
the different amount of TAMCN data withheld, are compiled in Table 3-27. 
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Table 3-27. Cross Validation Compiled Results for FSG 23 


FSG 

23* 

20% Data Withheld 
(36 of 180) 

15% Data Withheld 
(27 of 180) 

10% Data Withheld 
(18 of 180) 

Number of GRCs 

Percent in 

Number of GRCs 

Percent in 

Number of GRCs 

Percent in 


in Cl 

Cl 

in Cl 

Cl 

in Cl 

Cl 

LA 

2 of 36 

5% 

Oof 27 

0% 

2 of 18 

11% 

MS 

11 of36 

31% 

8 of 27 

30% 

5 of 18 

28% 


* Only Conflict Levels LA and MS are represented in this table. 


We are able to identify a trend in which, even at 90% bootstrapped confidence 
intervals constructed from only 50 replications, predicted values rarely remain inside that 
interval. This should remain as a cause for concern, since it is a clear indication of the 
lack of robustness we are detecting. 

When comparing the overall results of the cross validation for FSG 23, we can 
observe an FSG that is very susceptible to changes in the data. For the low-level assault 
phase, it is observed that any missing data would create a substantial detrimental effect 
on the minimum amount of reliability currently obtained on this FSG’s predicted CRC 
values. For the MS level, missing any more than 20% of the data could create a further 
rippling effect on the reliability of the predicted CRC values. 

3. Example Using FSC 5820, Radio Communications Equipment 

Another example of cross validation is used with FSC 5820, Radio and Television 
Communications Equipment (except airborne), where a total of 171 TAMCNs CEC 1 are 
available and we have originally 87 TAMCNs with each, ECC_LSs and ECC_MSs, and 
86 TAMCNs with each, ECC_LAs and ECC_MAs. 

The resulting figures and tables for this cross validation have also been omitted 
here, but can be found in Appendix J. The compiled results have been properly tabulated 
and are discussed at the end of this section. 

The effects observed when randomly withholding 20% (17 of the 87) of the 
available data and perfonning an OLS regression, with the response variable as ECC LA, 
show that the regression fit is valid with all of the numeric predictor terms’ resulting 
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p-value far below 0.1. When we compare the resulting predicted CRC values to the 
previously established confidence intervals, we are able to identify that only one of the 87 
TAMCNs, with data that were randomly selected and withheld, were predicted values 
that remained inside the mentioned interval. 

By randomly withholding only 15% (13 of 87) and 10% (9 of 87) of the data for 
ECC_LA, the regression fits remain perfectly valid with all the regression terms at very 
low p-values of at most 0.0012. Nevertheless, in both cases, we do not see an 
improvement when we compare the resulting predicted CRC values to the previously 
established confidence intervals, since only one of the 87 TAMCNs remained inside the 
mentioned interval. 

The effects observed when randomly withholding 20% (17 of the 87 withheld) of 
the available data and performing an OLS regression with the response variable as 
ECC_LS, show that the regression fit becomes invalid with all of the regression terms, 
hovering around 0.9. Comparing the resulting predicted CRC values to the previously 
established confidence intervals, 6 of the 17 TAMCNs were predicted values that 
remained inside the interval. 

When we randomly withhold only 15% and 10% of the data for ECC_LS, the 
regression fits remain invalid and all of the regression terms obtain p-values around 0.8. 
Yet, when we compare the resulting predicted CRC values to the previously established 
confidence intervals, on average, 40% remained inside the mentioned interval. 

Similar results are observed when randomly withholding the same depicted 
amounts of data throughout the rest of this analysis for ECC MAs and ECC MSs. The 
rest of the results for FSC 5820, at every level and phase of conflict and with the different 
amount of TAMCN data withheld, are compiled in Table 3-28. Also, as mentioned for 
FSC 2320, though we have established a flexible 90% confidence interval, there are only 
two models—in this case, CRC LS and CRC MS—that are able to show predicted 
values inside this interval, but only for 67% of the time. It is also evident that in the case 
of CRC LA, though showing better results for FSC 5820 compared to the results of 
FSC 2320, the regression fit’s perfonnance is unfavorable and does not provide any 
predicted values inside the above-mentioned confidence interval. 
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Table 3-28. Cross Validation Compiled Results for FSC 5820 


FSC 

5820 

20% Data Withheld 
(17 of 87) 

15% Data Withheld 
(13 of 87) 

10% Data Withheld 
(9 of 87) 

Number of CRCs 
in Cl 

Percent in 
Cl 

Number of CRCs 
in Cl 

Percent in 
Cl 

Number of CRCs 
in Cl 

Percent in 
Cl 

LA 

1 of 17 

6% 

1 of 13 

8% 

1 of 9 

11% 

LS 

5 of 17 

29% 

4 of 13 

31% 

6 of 9 

67% 

MA 

1 of 17 

6% 

3 of 13 

23% 

3 of 9 

33% 

MS 

7 of 17 

41% 

7 of 13 

54% 

6 of 9 

67% 


When comparing the overall results of the cross validation for FSC 5820, we can 
observe an FSC that has more consistent results, based on the greater availability of 
historical data. Although the regression fits are not perfectly valid, with some resulting in 
overwhelmingly invalid fits, we can see that LA remains unchanged through the process 
of comparing percentages of data missing. We can assume this FSC, and level and phase 
of conflict, to be lacking enough data to provide a sufficiently reliable regression model 
and predicted CRCs. 

The approach of gradually withholding data presented in this section, though 
labor-intensive, has the intention to demonstrate a method for exploring amount of data 
sufficiency. This is done in order to provide a means of identifying which amount of data 
would be necessary to replicate an efficient and valid model. In addition, from these few 
observations, it is evident that individual tests and studies seem to be necessary for every 
FSC and FSG when trying to pinpoint specific amounts of data required, if the intention 
is to obtain the most reliable and stable models for predicting CRCs and GRCs. We have 
also observed that every CRC and FSG reacts differently to the amount of data missing, 
not just because of the inherent equipment differences they are identified with, but also 
by the different levels and phases of conflict to which they apply. Improvements in 
consistency and accuracy in all types of data collected would be the overarching solution. 
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IV. CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY OF FINDINGS 

A full effort was made to replicate the procedure employed in the source code 
provided in the JMP file, but the same exact resulting values of the calculations, as 
presented in an early version of a CARF study draft report, were not obtained. The results 
of our calculations point to inconsistent models with substantially less than desirable 
regression fits. 

1. ECC Sensitivity 

We generate 50 bootstrapped replications of the historic usage data to obtain a 
distribution of ECCs for those TAMNs with available usage data. We show that based on 
the individual level and phase of conflict, many ECCs are significantly sensitive to 
minimal changes in the data with which they are calculated. We find that 12 of low level 
ECCs and 140 of medium level ECCs are not significantly different from zero. The least 
sensitive was the ECC_LAs, where 82 of 611 showed high sensitivity, while the most 
sensitive was the ECC_MSs, with 357 of 514 being highly sensitive. 

2. GLM and OLS Comparison 

We were unable to replicate the GLM regressions outlined in an early version of a 
CARF study draft report. Our results expressed that the fit of these models did not 
provide the validity necessary to predict reliable CRC or GRC values. Tenn significance 
in every one of these models was less than favorable. 

In comparison, an OLS approach had vast improvements in term significance and 
model validity. In most cases, we were able to obtain, on average, a 20% improvement in 
generating valid models across the levels and phases of conflict, which progressed into an 
increase in the number of CRCs and GRCs produced. For example, when observing 
CRCs for MSs, the increase goes from 3 to 586, and in GRCs for MAs, the increase goes 
from 59 to 707. No validation was performed in the entire OLS process. Employing OLS 
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regressions is representative of mainly exploratory intentions. Yet, the results obtained 
emphasize the need for further methodology research. 


B. RECOMMENDATIONS AND AREAS FOR FURTHER RESEARCH 

We emphasize that accurate CARF calculations provide the planning flexibility to 
meet requirements of any conflict and operational plan. It is imperative to understand that 
CARFs are an indispensable tool in determining WRM stocks. 

1. Bootstrapping 

The bootstrapping approach and analysis was only performed over casualties and 
only on 50 replications. Further simulation could be performed following the same 
presented approach over casualties with more than 1,000 replications. This could yield a 
more fine-tuned method to further identify which parts of the data are more sensitive to 
greater fluctuations. In addition, distributions of CRCs and GRCs could be attained by 
performing that analysis on each bootstrapped replication. 

2. Regression Analysis 

Some of the regression analysis perfonned for this thesis was done within JMP in 
order to replicate the proposed procedures. Other regression analysis was performed in 
SAS statistical software, which is the case of the OLS results presented. We recommend 
performing a comprehensive validation of each of the regression models built, to include 
analyzing the residuals to detennine normality, homoscedasticity, and level of 
autocorrelation. 
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APPENDIX A 


Screen capture of the JMP file that contains both the data-table, as well as the 
scripts to perform most of the necessary calculations. The first 39 rows of 6,137 are 
shown. 
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Commodity 

NOftMN 

NSN 

FSO 

CARF FSC 

FSO 

FSC 

FSO 

- R^wsadCARF 

8» 

i 

A00007G 

AOOOC 

A 

AOOOO INVENTORY ONLY 

9999014997305 

99 

9999 

99 

9999 

99 
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APPENDIX B 


Sample of Federal Supply Code (FSC) tables and classification list from 


http://www.dispositionservices.dla.mil/asset/fsclist.html. 


Aircraft and Airframe Structural Components 

1510 Aircraft, Fixed Wing 
1520 Aircraft, Rotary Wing 
1540 Gliders 
1550 Drones 

1560 Airframe Structural Components 
Aircraft Components and Accessories 
1610 Aircraft Propellers and Components 

1615 Helicopter Rotor Blades, Drive Mechanisms and Components. 

1620 Aircraft Landing Gear Components 
1630 Aircraft Wheel and Brake Systems 

1650 Aircraft Hydraulic, Vacuum, and De-icing System Components 

1660 Aircraft Air Conditioning, Heating, and Pressurizing Equipment 

1670 Parachutes; Aerial Pick Up, Delivery, Recovery Systems; and Cargo Tie Down 

Equipment 

1680 Miscellaneous Aircraft Accessories and Components 

Aircraft Launching, Landing, and Ground Handling Equipment 

1710 Aircraft Landing Equipment. 

1720 Aircraft Launching Equipment 

1730 Aircraft Ground Servicing Equipment 

1740 Airfield Specialized Trucks and Trailers 

1810 Space Vehicles 

1820 Space Vehicle Components 

1830 Space Vehicle Remote Control Systems 

1840 Space Vehicle Launchers 

1850 Space Vehicle Handling and Servicing Equipment 
1860 Space Survival Equipment 

Ships, Small Craft, Pontoons, and Floating Docks 

1905 Combat Ships and Landing Vessels 
1910 Transport Vessels, Passenger and Troop 
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1915 Cargo and Tanker Vessels 

1920 Fishing Vessels 

1925 Special Service Vessels 

1930 Barges and Lighters, Cargo 

1935 Barges and Lighters, Special Purpose 

1940 Small Craft 

1945 Pontoons and Floating Docks 

1950 Floating Drydocks 
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APPENDIX C 


From Section A, Chapter III: FSCs with ECCLSs and MSs that are outside of the 
sensitivity ratio criteria. 


Table C-l. ECC LSs and MSs that are outside of the sensitivity ratio criteria 


FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

B00157B 

2330 

Z BACKSCATTER RUGGEDIZED TRAILER (ZBRT) 

0 


E08567K 

2350 

ASSAULT AMPHIBIOUS VEHICLE, RECOVERY 

0 


E13787K 

2350 

RECOVERY VEHICLE, FT, HEAVY, W/EQUIP 

0 


A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


A23067G 

2355 

GROUND SENSOR SURVEILLANCE VEHICLE (GSSV) 

0 


A04997G 

5895 

DIGITAL TECHNICAL CONTROL (DTC) FACILITY 

0 


A04997G 

5895 

DIGITAL TECHNICAL CONTROL (DTC) FACILITY 

0 


A21797G 

5820 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


5820 

A00757G 

ARCHIVED TAMCN 

0 


5895 

A00917G 

VIDEO SCOUT REMOTE VIDEO EXPLOITATION 
TERMINAL (RVET) 

0 


5998 

A00917G 

VIDEO SCOUT REMOTE VIDEO EXPLOITATION 
TERMINAL (RVET) 

0 

0.0009 

5998 

A00917GA 

VIDEO SCOUT REMOTE VIDEO EXPLOITATION 
TERMINAL (RVET) 

0 

0.0006 

5895 

A02857G 

DISMTD DIGITAL AUTOMATED COMM TERMINAL 
(DDACT) 

0 

0.0004 

5895 

A04997G 

DIGITAL TECHNICAL CONTROL (DTC) FACILITY 

0 

0.0048 

5895 

A04997G 

DIGITAL TECHNICAL CONTROL (DTC) FACILITY 

0 

0.0072 

5820 

A09187G 

RADIO SET, SATELLITE, TACTICAL, PORTABLE 

0 

0.0007 

5820 

A09187G 

RADIO SET, SATELLITE, TACTICAL, PORTABLE 

0 

0.0009 

5820 

A09187GB 

RADIO SET, SATELLITE, TACTICAL, PORTABLE 

0 

0.0009 

5820 

A09187GC 

RADIO SET, SATELLITE, TACTICAL, PORTABLE 

0 

0.0009 

5820 

A09187GD 

RADIO SET, SATELLITE, TACTICAL, PORTABLE 

0 

0.0009 

6150 

A09207G 

SATELLITE COMMUNICATION 

0 

0.0008 

5985 

A13807G 

ANTENNA, COMMUNICATION, TRLR MTD, LTWT 

0 

0.007 

5985 

A13807G 

ANTENNA, COMMUNICATION, TRLR MTD, LTWT 

0 

0.0057 

5985 

A13807GA 

ANTENNA, COMMUNICATION, TRLR MTD, LTWT 

0 

0.0094 

5985 

A13807GB 

ANTENNA, COMMUNICATION, TRLR MTD, LTWT 

0 

0.0081 

5895 

A19587G 

KIT, MAINT, ELECTRONIC EQUIPMENT 

0 

0.0053 

5820 

A20427G 

RADIO SET, HIGH FREQUENCY, MANPACK 

0 

0.0001 

5820 

A20427G 

RADIO SET, HIGH FREQUENCY, MANPACK 

0 

0.0001 

5820 

A20427GA 

RADIO SET, HIGH FREQUENCY, MANPACK 

0 

0.0001 

5820 

A21717G 

RADIO SET, VEHICULAR 

0 

0.0022 

5820 

A21717GI 

RADIO SET, VEHICULAR 

0 

0.0015 

5820 

A21717GK 

RADIO SET, VEHICULAR 

0 

0.0022 
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FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

5820 

A21717GL 

RADIO SET, VEHICULAR 

0 


5820 

A21797G 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


5820 

A21797G 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


5820 

A21797G 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


5820 

A21797G 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


5820 

A21797G 

TERMINAL, RADIO, TROPO SCATTER, DIGITAL 

0 


2355 

A23067G 

GROUND SENSOR SURVEILLANCE VEHICLE (GSSV) 

0 


2355 

A23067G 

SENSOR SYSTEM, MONITOR, MOBILE 

0 


5411 

A23362B 

SHELTER, 20FT, EMI, MAINT COMPLEX 

0 

0.001 

5411 

A23382B 

SHELTER, 10FT, RIGID, MAINT COMPLEX 

0 


5805 

A25057G 

SWITCHBOARD, TELEPHONE, AUTOMATIC 

0 


5895 

A25357G 

DATA NETWORK, TACTICAL (GATEWAY) 

0 


5895 

A25357G 

DATA NETWORK, TACTICAL (GATEWAY) 

0 


5895 

A28087G 

TEST SET, OPTICAL COMMUNICATIONS 

0 


1550 

A32527G 

UAV SYSTEM, DRAGON EYE 

0 

0.001 

1550 

A32527G 

UAV SYSTEM, RAVEN 

0 


6625 

A70057G 

ANLYZER, SPECTRUM 

0 


6625 

A70057G 

ANLYZER, SPECTRUM 

0 


6625 

A70057G 

ANLYZER, SPECTRUM 

0 


6625 

A70057G 

ANLYZER, SPECTRUM 

0 


6625 

A70057G 

ANLYZER, SPECTRUM 

0 

0.0001 

6625 

A70057GA 

ANLYZER, SPECTRUM 

0 


6625 

A70097G 

ANALYZER, SPECTRUM, HAND HELD, CREW 

0 


6625 

A70097G 

ANALYZER, SPECTRUM, HAND HELD 

0 


6625 

A70257G 

COUNTER, ELECTRONIC 

0 


6625 

A70257G 

COUNTER, ELECTRONIC 

0 


6625 

A70257G 

COUNTER, ELECTRONIC 

0 


6625 

A70257G 

20 GHZ CW FREQUENCY COUNTER 

0 



A70377G 

OSCILLOSCOPE 

0 


6930 

A70467G 

GENERATOR, SIGNAL 

0 


6930 

A70467G 

GENERATOR, SIGNAL 

0 


6930 

A70467G 

40 GHZ SIGNAL GENERATOR 

0 


6930 

A70527G 

1 GHZ SIGNAL GENERATOR 

0 

0.0011 

6930 

A70597G 

SOIL RESISTIVITY TESTER 

0 


6930 

A70847G 

LOCAL AREA NETWORK TEST SET 

0 


6930 

A70847GA 

ANALYZER, NETWORK 

0 

0.0021 

6625 

A70867G 

OPTICAL TIME DOMAIN REFLECTOMETER (OTDR) 

0 

0.001 

6625 

A70867G 

OPTICAL TIME DOMAIN REFLECTOMETER (OTDR) 

0 


6625 

A70867G 

OTDR 

0 


5180 

A79002E 

ELECTRONIC TOOL KIT 

0 

0.0001 

5180 

A79107G 

TOOL KIT, FIBER OPTIC, GP 

0 


5180 

A79107G 

GENERAL PURPOSE FIBER OPTIC TOOL KIT 

0 


5895 

A79557G 

MICRO MINIATURE REPAIR STATION (PACE KIT) 

0 

0.003 

5895 

A79557G 

MICRO MINIATURE REPAIR STATION (PACE KIT) 

0 

0.0034 

6080 

A79657G 

KIT, CONNECTOR, FIBER OPTIC 

0 

0.0231 

5810 

A80447G 

LIMITED MAINT SPARE PARTS KIT 

0 


2330 

B00157B 

Z BACKSCATTER RUGGEDIZED TRAILER (ZBRT) 

0 


2330 

B00157B 

VAN, Z BACKSCATTER 

0 

0.0014 
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FSC 

TAMCN 

NOMENCLATURE 

Interval 

Lower 

Bound 

Interval 

Upper 

Bound 

4210 

B00457B 

EXPEDITIONARY FIRE SUPPRESSION SYSTEM 

0 


3810 

B04467B 

AIR MOBILE CRANE, RT, HYDRAULIC, LT (SLEP) 

0 


3810 

B04467B 

AIR MOBILE CRANE, RT, HYDRAULIC, LT (SLEP) 

0 


3810 

B04467B 

AIR MOBILE CRANE, RT, HYDRAULIC, LT (SLEP) 

0 


8110 

B05717B 

DRUM, FABRIC, COLLAPSIBLE, 500 GAL. CAP. 

0 


8110 

B05717B 

DRUM, FABRIC, COLLAPSIBLE, 500 GAL. CAP. 

0 

0.0011 

2430 

B05897B 

EXCAVATOR, COMBAT 

0 

0.003 

2430 

B05897B 

M9 ARMORED COMBAT EARTHMOVER 

0 


4210 

B06257B 

COMPRESSED AIR-FOAM SYSTEM, MOBILE 

0 


4930 

B06757B 

TACTICAL AIRFIELD FUEL DISPENSING SYSTEM 
(TAFDS) (FIRESTONE) 

0 

0.0101 

4930 

B06757B 

TACTICAL AIRFIELD FUEL DISPENSING SYSTEM 
(TAFDS) (FIRESTONE) 

0 

0.0101 

4930 

B11357B 

REFUELING SYSTEM, EXPEDIENT, HELO 

0 


1055 

B12987B 

KIT, LAUNCH, LINE CHARGE, TRLR-MTD 

0 


1055 

B12987B 

KIT, LAUNCH, LINE CHARGE, TRLR-MTD 

0 


4320 

B15707B 

EXPEDIENT REFUELING SYSTEM (ERS) 

0 


4320 

B15707B 

EXPEDIENT REFUELING SYSTEM (ERS) 

0 


5180 

B22602E 

TOOL KIT, PIONEER, ENGR SQUAD 

0 


5180 

B22602E 

PIONEER KIT (SQD) 

0 


2430 

B24627B 

TRACTOR, FT, MEDIUM (CATERPILLAR) 

0 


6630 

B26307B 

ANALYSIS SET, QUALITY, PURIFICATION, WATER 

0 

0.0081 

6630 

B26307B 

ANALYSIS SET, QUALITY, PURIFICATION, WATER 

0 

0.0091 

6630 

B26307B 

ANALYSIS SET, QUALITY, PURIFICATION, WATER 

0 

0.0081 

3431 

B26857B 

WELDING SHOP, MARINE CORPS TACTICAL 

0 


6625 

B70012G 

ANALYZER, ELECTRICAL PULSE 

0 


6625 

B70012G 

ANALYZER, ELECTRICAL PULSE 

0 

0.0181 

6625 

B70012G 

3-PHASE POWER ANALYZER 

0 

0.0163 

5855 

C00042E 

ULTRA HIGH INTENSITY MINIATURE ILLUMINATION 
SYSTEM 

0 

0.0001 

3530 

C60702T 

SEWING MACHINE, INDUSTRIAL, HVY DUTY, LEATHER 

0 


3530 

C60812T 

SEWING MACHINE, ZIGZAG, MED DUTY 

0 


3530 

C60812T 

SEWING MACHINE, ZIGZAG, MED DUTY 

0 


3530 

C61012T 

SINGLE NEEDLE KIT-MEDIUM WEIGHT 

0 


4910 

C70167B 

DYNAMOMETER, RUN-IN, 1800HP 

0 


4940 

C70252B 

TEST SET, HYDRAULIC, IN-LINE 

0 


4940 

C70252B 

TEST SET, HYDRAULIC, IN-LINE 

0 

0.0041 

4940 

C70252B 

TEST SET, HYDRAULIC, IN-LINE 

0 

0.0062 

4910 

C70722B 

TEST STAND, GEN/STARTER AND ALTERNATOR 

0 

0.0011 


25 

Scaled Deviance 

5 

0 

0 orig_fsg_la 

25 

Pearson Chi-Square 

5 

1.94933E-31 

3.89866E-32 orig_fsg_la 

25 

Scaled Pearson X2 

5 

1.94933E-31 

3.89866E-32 orig_fsg_la 

25 

Log Likelihood 


-0.80257565 

orig_fsg_la 

34 

Deviance 

0 

0 

orig_fsg_la 

34 

Scaled Deviance 

0 

0 

orig_fsg_la 

34 

Pearson Chi-Square 

0 

2.23489E-33 

orig_fsg_la 

34 

Scaled Pearson X2 

0 

2.23489E-33 

orig_fsg_la 
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